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1.  Introduction 


1.1  Background 

Maes  (1995)  proposed  a  continuum  of  robot  control  that  ranges  from  teleoperation  to  full 
autonomy.  Robots  can  be  placed  on  this  continuum  based  on  the  level  of  human-robot 
interaction  that  is  required.  Teleoperation  is  the  lowest  level  of  automation  on  the  continuum 
because  it  requires  the  most  intervention  from  the  robot  operator.  A  teleoperated  robot  is  totally 
under  the  control  of  the  operator  who  uses  a  joystick  or  other  control  device  to  command  the 
robot.  This  requires  constant  interaction  between  the  robot  and  the  operator.  Teleoperation  is 
not  the  ideal  solution  for  all  situations.  Many  teleoperation  tasks  are  repetitive  and  boring  and 
the  work  requires  constant  attention  by  the  operator. 

Semi-autonomous  control  (often  called  supervisory  control)  requires  the  operator  to  provide  an 
instruction  or  portion  of  a  task  that  can  safely  be  performed  by  the  robot  on  its  own.  Two  types 
of  semi-autonomous  control  are  often  identified — shared  control  (also  called  continuous 
assistance)  and  control  trading.  Shared  control  requires  the  teleoperator  to  delegate  a  task  for  the 
robot  or  to  accomplish  it  via  direct  control  of  the  robot.  If  the  operator  delegates  control  to  the 
robot,  he  or  she  must  still  monitor  the  robot  to  ensure  that  it  is  performing  the  task  correctly.  A 
guarded  teleoperated  robot  can  be  placed  on  this  portion  of  the  continuum  because  it  has  the 
ability  to  sense  and  avoid  obstacles  but  will  otherwise  navigate  as  driven,  like  a  robot  under 
manual  teleoperation.  During  control  trading,  the  human  only  interacts  with  the  robot  to  give  it  a 
new  command  or  to  interrupt  it  and  change  its  orders.  A  line-following  robot  can  be  placed  on 
this  portion  of  the  continuum  if  it  simply  follows  something  painted,  embedded  or  placed  on  the 
floor  and  does  not  have  the  ability  to  circumnavigate  obstacles  on  its  own. 

After  receiving  instructions,  autonomous  robots  (those  that  perform  tasks  independently  and/or 
have  the  capacity  to  choose  goals  for  themselves)  operate  under  all  reasonable  conditions 
without  recourse  to  an  outside  operator  and  can  handle  unpredictable  events  (Haselager,  2005). 
Huang,  et  al.  (2005)  state  that  a  fully  autonomous  robot  has  the  ability  to  gain  information  about 
the  environment,  work  for  an  extended  period  without  human  intervention,  move  itself  through 
its  environment  without  human  assistance,  and  avoid  situations  that  are  harmful  to  people, 
property,  or  itself  unless  those  are  part  of  its  design  specifications.  An  autonomously  guided 
robot  can  be  placed  on  this  portion  of  the  continuum  because  it  knows  at  least  some  information 
about  where  it  is  and  how  to  reach  waypoints  (Murphy,  2000).  Knowledge  of  its  current  location 
is  determined  by  using  sensors  such  as  lasers  and  global  positioning  systems  (GPS).  Positioning 
systems  determine  the  location  and  orientation  of  the  platform,  so  the  robots  can  plan  a  path  to 
their  next  waypoints  or  goals.  Autonomous  robots  make  their  own  choices  instead  of  following 
goals  set  by  other  agents.  A  robot  that  is  capable  of  goal  generation  can  also  be  placed  on  this 
portion  of  the  continuum. 
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The  incorporation  of  autonomous  robotic  systems  into  military  schema  involves  more  than  just 
effective  system  design.  It  is  also  important  for  designers  to  understand  which  potential  robotic 
system  autonomous  capabilities  match  or  surpass  human  abilities  and  when,  during  scenario 
accomplishment,  the  human  needs  assistance  or  is  overloaded.  This  knowledge  is  needed  in 
order  to  leverage  autonomous  robotic  behaviors  for  optimal  performance.  Thus  the  goal  is  to 
optimize  the  human  and  robot  roles  in  a  task.  To  do  this,  robot  control  competencies  and 
inefficiencies  must  first  be  identified  and  then  they  must  be  understood  in  relation  to  specific 
task  performance.  Trade-offs  among  levels  of  autonomy  must  be  identified.  For  example,  an 
autonomous  system  may  have  a  slower  reaction  time  to  difficult  problems  than  a  system  being 
teleoperated  by  a  human  but  latency  involved  in  teleoperation  could  render  the  human’s  reaction 
time  ineffective.  Also,  moderators  of  robot  control  performance  must  be  identified.  For 
example,  workload  is  also  an  important  moderator  of  human  task  performance.  The 
effectiveness  of  a  control  system  is  likely  to  depend  a  great  deal  on  situational  task  demands. 

The  requirement  for  the  operator  to  perform  simultaneous  tasks  such  as  controlling  multiple 
robots  or  performing  local  security  tasks  could  have  a  detrimental  effect  on  human  intervention 
in  robotic  task  outcomes  because  of  human  availability  at  a  certain  point  in  time  and  because  of 
cognitive  overload. 

Many  studies  have  demonstrated  that  operators’  situation  awareness  (SA)  was  higher  when  they 
were  controlling  robots  with  semiautonomous  or  autonomous  capabilities  (Chen  et  al.,  2008; 
Dixon  et  al.,  2003;  Luck  et  al.,  2006).  However,  increased  autonomy  is  not  a  panacea.  In  a  two- 
year  study  of  a  collaborative  human-robot  system  Stubbs  et  al.  (2007)  found  that  as  autonomy 
increased,  users’  inability  to  understand  the  reasons  for  the  robot’s  actions  disrupted  the  creation 
of  common  ground.  Also,  many  fear  that  providing  more  and  more  autonomy  to  armed  robots 
could  result  in  collateral  damage  or  fratricide  (Singer,  2009). 

1.2  Objective 

The  goal  of  this  research  was  to  examine  the  effectiveness  of  different  levels  of  automation 
(teleoperation,  semi-autonomous,  and  autonomous)  on  robot  control  in  reconnaissance  missions 
when  the  robotic  operator  is  fully  engaged  in  additional  high  cognitive  load  activities.  Our 
hypothesis  was  that  full  autonomy  would  be  the  most  effective  level  of  autonomy  for  a  robotic 
reconnaissance  mission  when  the  robotic  operator  is  fully  engaged  in  a  high  cognitive  load 
activity. 

1.3  Overview  of  the  Experiment 

This  study  was  a  cooperative  research  effort  between  the  U.S.  Army  Research  Laboratory 
(ARL)/Human  Research  and  Engineering  Directorate  (HRED)  and  the  Space  and  Naval  Warfare 
Systems  Center  San  Diego  (SSC  Pacific).  It  was  an  investigation  of  the  effect  of  progressive 
levels  of  automation  on  robotic  reconnaissance  task  performance.  It  took  place  at  Fort  Benning, 
GA.  Thirty  Soldiers  from  the  Officer  Candidate  School  (OCS)  and  the  Warrior  Training  Center 
(WTC)  participated  in  the  study.  After  training  on  the  operation  of  the  robotic  system,  each 
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Soldier  completed  reconnaissance  exercises  using  three  different  levels  of  robotic  automation 
(teleoperated,  semi-autonomous  and  autonomous).  During  the  exercises,  Soldiers  responded  to 
requests  for  information  regarding  situation  and  mission  awareness.  The  terrain  and  hazards 
were  counter-balanced  along  with  the  automation  level  to  control  for  the  effect  of  learning. 
Automation  level  and  usability  were  evaluated  based  on  objective  performance  data,  data 
collector  observations,  and  Soldier  questionnaires. 


2.  Method 


2.1  Participants 

Thirty  Soldiers  from  the  OCS  and  WTC  participated  in  the  study.  The  OCS  participants 
included  Soldiers  with  prior  enlisted  service  with  a  variety  of  backgrounds  and  experience  levels 
as  well  as  those  just  coming  into  the  Army  from  college.  The  WTC  Soldiers  consisted  of 
enlisted  Soldiers  with  the  rank  of  E-4  through  E-6  serving  as  instructors  for  the  pre-Ranger  and 
air  assault  courses. 

2.1.1  Pretest  Orientation 

The  Soldiers  were  given  an  orientation  on  the  purpose  of  the  study  and  what  their  participation 
would  involve.  They  were  briefed  on  the  objectives  and  procedures,  as  well  as  on  the  robot. 
They  were  also  told  how  the  results  would  be  used  and  the  benefits  the  military  could  expect 
from  this  investigation.  Any  questions  the  subjects  had  regarding  the  study  were  answered. 

2.2  Apparatus  and  Instruments 

2.2.1  SSC  Pacific  Robot 

The  SSC  Pacific  robot  (figure  1)  is  an  iRobot  PackBot  Scout,  equipped  with  a  1st  generation 
Navigator  payload.  The  payload  contains  a  sensor  suite  that  includes  an  inertial  measurement 
unit  (IMU),  a  gyroscope,  a  compass,  GPS,  and  a  360°  ladar.  The  IMU,  gyroscope,  compass,  and 
GPS  are  used  for  positioning,  localization,  and  navigation,  while  the  ladar  is  used  for  obstacle 
avoidance  and  mapping.  In  addition,  the  payload  contains  a  processor  that  runs  the  Autonomous 
Capability  Suite  (ACS)  that  includes  all  the  autonomous  behaviors  onboard  the  robot  and 
Freewave  radio  to  communicate  with  the  operator  control  unit.  The  scalability  and  configurable 
framework  of  ACS  provided  the  key  sliding  autonomy  feature  needed  to  conduct  this 
experiment;  the  operator  was  able  to  change  the  level  of  autonomy  onboard  the  robot  by  a  press 
of  a  button  on  the  operator  interface. 
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Figure  1.  SSC  Pacific  robot  with  1st  generation 
navigator  payload. 


2.2.2  Operator  Interface 

The  operator  interface  used  to  control  the  system  was  based  on  SSC  Pacific’s  Multi-Robot 
Operator  Control  Unit  (MOCU).  An  example  screenshot  of  the  interface  is  found  in  figure  2. 
The  robot’s  location,  driven  path,  goal  points,  and  sensor  data  (i.e.,  map  data)  were  overlaid  on 
an  aerial  image.  Real-time  video  from  the  robot  was  also  displayed. 


Figure  2.  Snapshot  of  MOCU  interface. 


Various  button  and  joystick  controls  were  also  provided  to  send  drive  commands  to  the  robot,  as 
well  as  to  turn  on  and  off  the  robot’s  behaviors.  This  allowed  the  operator  to  control  the  sliding 
autonomy  available  on  the  robot.  Three  levels  of  autonomy  were  achieved  by  turning  on  and  off 


4 


robot  behaviors.  All  behaviors  were  turned  off  during  teleoperation  so  that  the  operator  was  in 
complete  control  of  all  robot  actions  and  the  operator  had  to  draw  the  building  map  and  mark  the 
location  of  items  of  interest  by  hand.  Semi-autonomy  was  achieved  by  using  the  obstacle 
avoidance,  mapping,  and  retum-to- start  functions  available  on  the  robot.  This  function  assisted 
the  robot  operator  during  driving  by  automatically  avoiding  obstacles,  self  directing  itself  to  open 
spaces,  building  a  map  of  the  building  for  the  operator,  and  returning  to  the  start  location  on  its 
own  when  the  operator  completed  his/her  mission.  Full  autonomy  was  achieved  by  turning  on 
the  efficient  exploration  behavior  in  addition  to  all  the  behaviors  included  under  semi-autonomy. 
Table  1  shows  the  behaviors  used  for  each  level  of  autonomy. 


Table  1.  Autonomous  behaviors  used  for  each  level  of  autonomy. 


Self-Directing 

Obstacle  Avoidance 

Mapping 

Return-to-Start 

Exploration 

Teleoperated 

No 

No 

No 

No 

No 

Semi-autonomous 

Yes 

Yes 

Yes 

Yes 

No 

Fully-autonomous 

Yes 

Yes 

Yes 

Yes 

Yes 

The  MOCU  software  was  developed  independent  of  the  hardware  unit,  and  thus  can  be  run  on 
any  computer.  A  commercial  grade  off-the-shelf  (COTS)  laptop  was  used  in  the  experiment  to 
run  MOCU.  A  Microsoft  Xbox  360  wireless  (figure  3)  controller  was  used  to  provide  a 
joystick/button  interface. 


LB  Left  did  Mouse 


Back  Return  to  Start 


Left  Joystick: 
Robot  Steermf 


c 

o 

0, 


Hat-switch: 


Stop 

Right  45  deg 
Right  90  deg 
Right  155  deg 


Q 

o 

vO 


RB:  Right  dick  Mouse 


X:  Fully  autonomous  Mode 
YtTeleopMode 

B  Semi-autonomous  Mode 
A:  Stop 

Start:  Take  Snapshot 

Right  Joystick:  Mouse 
Button:  Left  did  Mouse 


LT:  Zoom  Out 


RT:  Zoom  In 


Turn  180  deg 
Left45deg 
left  90  deg 
leftl55deg 


Figure  3.  Microsoft  Xbox  360  wireless  controller. 
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2.2.3  Building  Reconnaissance  Course 


The  building  reconnaissance  course  (figure  4)  was  located  at  Molnar  military  operations  in  urban 
terrain  (MOUT)  site,  Ft  Benning,  GA.  It  consisted  of  three  one-story  buildings  that  were  similar 
in  size  but  with  different  floor  plans.  Soldier  operators  were  located  out  of  the  line  of  sight  of  the 
robot  in  a  stationary  position  inside  a  separate  building. 


Figure  4.  Building  reconnaissance  course. 


Various  items  were  placed  in  the  rooms  of  each  building  and  arranged  to  represent  a  specific 
enemy  activity.  Building  A  was  configured  to  represent  a  chemical  weapons  factory,  building  B 
was  configured  to  represent  a  staging  area  for  a  terrorist  attack  against  the  United  States,  and 
building  C  was  configured  to  represent  a  prisoner  of  war  (POW)  interrogation  facility.  Figures 
5-7  show  the  floor  plans  and  placement  of  items  for  each  building. 


Item_ Description 


1 

AK47 

2 

AK47 

3 

AK47 

4 

RPG 

5 

Cardboard  box 

6 

6v  Battery 

7 

NBC  Mask 

8 

NBC  doves 

9 

Eye  Protection 

10 

120  mm  Tank  Round 

11 

120  mm  Tank  Round 

12 

120  mm  Tank  Round 

13 

105  mm  Arty  Round 

14 

105  mm  Arty  Round 

15 

Funnel 

16 

Chemical  Sprayer 

17 

Container  with  Chemical  warning 
label 

18 

Container  with  Chemical  warning 
label 

19 

Cell  Phone 

Figure  5.  Building  A:  chemical  weapons  factory. 
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Item 

Description 

Diet  Pepsi  Caffeine  Free  Can 

1 

bomb 

2 

Diet  Coke  bomb 

3 

Diet  Pepsi  Can  bomb 

4 

Land  Mine  Grey 

5 

Land  Mine  Green  and  Tan 

6 

land  Mine  Tan 

7 

Cardboard  box 

8 

picture  of  statue  of  liberty 

9 

Suicide  bomber  vests 

10 

Suicide  bomber  vests 

11 

Suicide  bomber  vests 

12 

Briefcase  bomb 

13 

Map  with  New  York  circled 

14 

Timer 

15 

6v  Battery 

16 

Wire 

17 

Initiabon  Device 

18 

PDA 

19 

Cell  Phone 

20 

Pipe  Bomb 

r 21 

82mm  Mortar  round 

22 

122mm  Projectile 

23 

130mm  Rocket  round 

Figure  6.  Building  B:  terrorist  staging  area. 


_ _ Description 

1  Cardboard  bout _ 

2  PDA 

3  ’Fan 

4  AK47 

5  AK47 

6  AK47 

7  RPG 

8  12v  Battefy 

9  Jumper  cables 

_ 10  Chair 

11  Table 

12  Lamp 

13  Wire 

14  Rope 

15  Wire 
16,17  ACU  unifonn 

18  Beddina 

19  Bloody  rags 

20  Tortured  dead  body 

21  Sandbag  over  head 

22  Chair 

23  Bedding 


Figure  7.  Building  C:  POW  interrogation  facility. 

2.2.4  The  National  Aeronautics  and  Space  Administration- Task  Load  Index  (NASA-TLX) 

The  NASA-TLX  is  a  subjective  workload  assessment  tool  that  allows  subjective  workload 
assessments  on  operators  working  with  various  human-machine  systems  (Hart  and  Staveland, 
1988).  It  uses  a  multi-dimensional  rating  procedure  that  derives  an  overall  workload  score  based 
on  a  weighted  average  of  ratings  on  six  subscales.  These  subscales  include  mental  demands, 
physical  demands,  temporal  demands,  own  performance,  effort,  and  frustration.  It  can  be  used  to 
assess  workload  in  various  human-machine  environments  such  as  aircraft  cockpits;  command, 
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control,  and  communication  workstations;  supervisory  and  process  control  environments; 
simulations;  and  laboratory  tests.  The  version  of  the  NASA-TLX  used  during  this  experiment 
was  presented  to  the  Soldiers  on  a  computer.  Definitions  of  each  scale  were  provided  on 
laminated  paper  so  the  participates  could  refer  to  it  as  they  were  providing  their  estimates  of  the 
workload  associated  with  each  type  of  small  unmanned  ground  vehicle  (SUGV)  control  on  the 
various  scales. 

2.2.5  Questionnaires 

A  demographic  questionnaire  was  administered  to  gather  information  concerning  demographic 
data,  robotic  experience,  and  physical  characteristics  that  might  have  had  an  effect  on  the 
participant’s  ability  to  operate  the  robot.  The  questionnaires  given  after  participation  in  the 
events  were  designed  to  elicit  Soldiers’  opinions  about  their  performance  and  experiences  with 
each  of  the  control  systems.  They  asked  the  Soldiers  to  rate  the  devices  on  a  7-point  semantic 
differential  scale  ranging  from  “extremely  good/easy”  to  “extremely  bad/difficult.”  A  post 
iteration  questionnaire  was  administered  to  each  Soldier  at  the  end  of  each  iteration  with  each 
level  of  autonomy,  and  an  end  of  experiment  questionnaire  was  administered  at  the  completion 
of  all  three  iterations. 

2.3  Procedures 

2.3.1  Soldier  Orientation 

Upon  arrival,  the  experiment  Soldiers  received  a  roster  number,  which  was  used  to  identify  them 
throughout  the  evaluation.  During  the  pretest  orientation,  the  Soldiers  were  given  an  operations 
order  that  explained  the  robotic  mission  that  they  would  be  undertaking  during  the  experiment. 

A  copy  of  the  operations  order  can  be  found  in  appendix  A  (FM  5-170,  1998).  It  was  explained 
to  them  that  upon  completion  of  training  they  would  complete  three  iterations  of  the  building 
reconnaissance  course  (one  with  each  level  of  autonomy)  according  to  the  matrix  in  table  2.  All 
instruments  that  were  used,  the  training  course,  and  the  building  reconnaissance  course  were 
explained  and  any  questions  the  Soldiers  had  concerning  the  experiment  were  answered.  They 
were  also  told  how  the  results  would  be  used  and  the  benefits  the  military  could  expect  from  the 
investigation.  Demographic  data,  as  well  as  data  concerning  their  Army  and  robotic  experience 
and  physical  characteristics,  were  also  taken  for  each  Soldier. 
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Table  2.  Order  of  treatments  and  lanes. 


Building  and  Iteration 

Roster  Number 

Bldg  A,  Trial  1 

Bldg  B,  Trial  2 

Bldg  C,  Trial  3 

1,  16 

T 

S 

A 

2,17 

A 

T 

S 

3,18 

S 

A 

T 

4,19 

T 

A 

S 

5,20 

S 

T 

A 

6,21 

A 

S 

T 

7,22 

S 

T 

A 

8,23 

A 

S 

T 

9,24 

T 

A 

S 

10,25 

S 

A 

T 

11,26 

T 

S 

A 

12,27 

A 

T 

S 

13,28 

S 

A 

T 

14,29 

A 

T 

S 

15,30 

T 

S 

A 

Note:  T  =  teleoperation;  S  =  semi-autonomous;  A  =  autonomous 


2.3.2  Demographics 

Demographic  data  were  taken  for  each  Soldier.  Data  concerning  their  physical  characteristics 
and  experience,  especially  their  knowledge  of  operating  remote  controlled  vehicles,  were 
included  in  the  demographic  data  sheet.  The  demographic  data  sheet  is  shown  in  appendix  B. 

2.3.3  Training 

Training  was  conducted  by  a  representative  of  SSC  Pacific.  Groups  of  three  Soldiers  were  given 
an  overview  of  the  robot  capabilities,  controls,  and  sensors,  and  the  three  autonomy  levels  were 
explained.  Then  each  Soldier  was  given  15  min  of  individual  hands-on  training  that  included 
practice  on  the  tasks  required  to  perform  the  building  reconnaissance  in  all  three  levels  of 
autonomy.  The  trainer  assessed  whether  the  Soldiers  were  capable  of  performing  the  tasks  in  all 
three  levels  of  autonomy  before  they  were  allowed  to  participate  in  the  study. 

During  training,  Soldiers  were  issued  a  battalion  level  operations  order  outlining  the 
reconnaissance  mission  they  were  supporting.  Each  Soldier  was  given  about  15  min  to  read  the 
order  prior  to  conducting  the  reconnaissance  course. 

2.3.4  Building  Reconnaissance  Course 

The  Soldiers  completed  the  building  reconnaissance  course  with  the  level  of  autonomy  and  the 
lane  order  assigned  in  table  2.  The  course  consisted  of  three  different  one- story  buildings  so  the 
Soldiers  could  reconnoiter  a  different  building  with  each  level  of  autonomy. 
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During  teleoperation  trials,  the  robot  was  manually  driven  using  the  video  feed  displayed  on  the 
MOCU  interface  and  left  joystick  gimbal  on  the  Microsoft  Xbox  360  wireless  controller  (figure 
3). 

During  the  semi-autonomous  trials,  the  robot’s  obstacle  avoidance,  mapping,  and  retum-to- start 
behaviors  were  turned  on,  and  the  robot  self-directed  itself  through  the  building  to  the  next  open 
space  available.  The  Soldier  was  able  to  stop  the  robot  from  self-directing  itself  and  take  control 
at  any  time  by  pressing  the  A  button  to  stop  the  robot  and  used  the  left  joystick  gimbal  to 
influence  the  robot’s  direction.  To  turn  on  the  robot’s  self-directing  behavior  again,  the  Soldier 
pressed  the  B  button.  The  obstacle  avoidance  feature  was  always  on  to  prevent  the  robot  from 
mnning  into  objects.  To  turn  obstacle  avoidance  off,  the  Soldier  had  to  switch  to  teleoperation 
mode  by  pressing  the  Y  button.  When  the  Soldier  decided  all  items  of  interest  were  found,  the 
BACK  button  was  pressed  on  the  X-box  controller  and  the  robot  returned  to  the  starting  location 
on  its  own. 

During  the  fully-autonomous  trials,  the  robot’s  obstacle  avoidance,  mapping,  and  retum-to- start 
behaviors  were  turned  on,  as  well  as  an  efficient  exploration  behavior  that  helped  the  robot  self 
direct  itself  in  places  it  had  not  driven  to  yet,  to  minimize  the  overall  time  to  explore  the  building 
while  maximizing  coverage  area.  As  in  the  semi-autonomous  mode,  the  Soldier  was  able  to  stop 
the  robot  at  any  time  by  pressing  the  A  button.  The  Soldier  could  also  switch  teleoperation  mode 
(by  pressing  the  A  button)  or  semi-autonomous  mode  (by  pressing  the  X  button).  Also  as  in  the 
semi-autonomous  mode,  the  Soldier  was  able  to  send  the  robot  back  to  the  starting  location  by 
pressing  the  BACK  button. 

For  all  three  trials  in  each  of  the  autonomy  levels,  the  Soldier  was  required  to  take  snapshots  of 
items  of  interest  found  and  label  them.  The  camera  was  in  a  fixed  position  facing  the  front  of  the 
robot  and  did  not  have  a  pan  and  tilt  capability.  In  order  to  take  a  snapshot,  the  Soldier  had  to 
maneuver  the  robot  into  a  position  facing  the  item.  To  label  the  snapshots,  the  Soldier  used  the 
right  joystick  gimbal  to  move  the  mouse  to  the  snapshot,  click  the  right  trigger  button  on  the 
X-box  controller  to  bring  up  the  list  of  labels,  and  then  click  the  left  trigger  button  to  select  the 
label. 

Soldiers  were  instructed  that  their  primary  tasks  were  to  provide  a  map  of  the  floor  plan  of  the 
building,  to  identify  objects  of  interest  (SA  task)  and  take  a  snapshot  of  them,  and  return  the 
robot  to  the  starting  point  as  quickly  as  possible  once  the  entire  building  had  been  reconnoitered 
and  mapped.  During  autonomous  and  semi-autonomous  trials,  the  system  automatically  mapped 
the  buildings  as  the  robot  maneuvered  through  them  and  snapshots  of  the  items  appeared  on  the 
map  in  the  location  they  were  taken.  During  teleoperated  trials,  Soldiers  were  instructed  to  draw 
a  scaled  map  by  hand  of  the  floor  plan  of  each  building  depicting  the  location  of  items  of 
interest. 
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The  Soldiers’  secondary  tasks  were  to  answer  questions  concerning  details  of  their  mission  based 
on  the  information  provided  in  the  operations  order,  and  to  identify  different  types  of  smoke 
grenades.  The  secondary  tasks  served  as  an  additional  cognitive  load.  Secondary  task  questions 
were  asked  at  2  min  intervals  throughout  the  reconnaissance.  Mission  details  could  be  found  in 
excerpts  of  the  operations  order  that  was  provided  to  them  during  each  trial.  They  were 
instructed  to  look  up  and  answer  questions  about  the  operations  order  only  if  doing  so  did  not 
interfere  with  performance  of  the  primary  tasks. 

A  data  collector  accompanied  the  robot  to  record  completion  times  and  driving  errors  (i.e., 
running  into  walls  or  objects  with  the  robot).  Another  data  collector  was  present  at  the  operator 
station  to  record  the  number  and  types  of  items  of  interest  identified  (SA  task),  and  to  ensure  the 
Soldiers  prioritized  their  focus  on  the  primary  task.  An  additional  data  collector  asked  the 
Soldiers  secondary  task  questions  at  two  minute  intervals  and  recorded  their  responses. 

Upon  the  completion  of  the  iteration,  the  Soldiers  were  provided  with  a  list  of  the  items  of 
interest  they  identified  and  were  asked  to  answer  situational  awareness  questions  concerning  the 
type  of  enemy  activity  that  had  taken  place  in  the  building  based  on  the  items  found  in  the 
building.  They  were  also  asked  to  fill  out  questionnaires  concerning  the  trial  they  just  completed 
and  to  fill  out  a  NASA-TLX  concerning  the  level  of  workload  experienced. 

2.3.5  End  of  Iteration  Questionnaire  Administration 

Questionnaires,  designed  to  assess  participants’  performance  and  experiences  with  each  level  of 
autonomy,  and  the  NASA-TLX  were  administered  to  each  Soldier  at  the  end  of  each  iteration. 
After  completing  the  course  with  each  level  of  autonomy,  the  Soldiers  completed  an  end  of 
experiment  questionnaire  that  compared  each  condition. 

2.4  Experimental  Design 

The  design  of  this  experiment  was  a  single  factor  repeated  measures  design  (table  2). 

2.4.1  Independent  variable 

•  Level  of  autonomy  (reconnaissance  course  events). 

2.4.2  Dependent  Variables 

•  Building  reconnaissance  course  completion  time. 

•  The  number  of  driving  errors  on  each  course. 

•  The  number  of  correct  objects  of  interest  found  on  each  course. 

•  Accuracy  of  maps  created  during  the  building  reconnaissance  course  (correct  number  of 
doors,  items  of  interest  in  correct  relation  to  each  other,  correct  number  of  rooms). 

•  S A  questionnaire  responses. 
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•  Number  of  secondary  task  questions  answered  correctly. 

•  NASA-TLX  scores  for  each  level  of  automation  after  each  course  is  completed. 

•  Data  collector  comments. 

•  Questionnaire  ratings  and  comments. 

2.5  Data  Analysis 

All  objective  data  collected  on  the  reconnaissance  courses  were  analyzed  using  repeated 
measures  analysis  of  variance  (ANOVA).  Follow-on  pair  wise  comparisons  were  done  using 
Holm’s  Bonferroni  procedure  to  control  for  family-wise  error  rates  (Holm,  1979).  Partial  eta 
squared  (rfp),  an  index  of  effect  size,  was  computed  for  each  ANOVA.  Iteration  effects  were 
controlled  through  the  counterbalanced  order  of  the  experimental  design  (table  2).  Soldier 
questionnaire  data  were  analyzed  using  descriptive  statistics  on  the  subjective  ratings. 


3.  Results 


3.1  Demographics 

The  Soldiers  ranged  in  rank  from  E4  to  E6.  The  average  age  of  the  Soldiers  was  28  years  and 
the  average  time  in  service  was  68  months.  None  of  the  Soldiers  had  any  prior  experience  in 
teleoperating  a  ground  unmanned  robot.  Detailed  responses  to  the  demographics  questionnaire 
are  available  in  appendix  B . 

3.2  Training 

The  participants  rated  the  training  as  being  very  good  for  all  levels  of  autonomy.  They  indicated 
that  the  hardest  task  to  learn  was  mapping  the  building  in  the  teleoperation  mode.  Driving  the 
robot  was  the  easiest  task  to  learn  in  all  three  levels  of  autonomy.  Detailed  responses  to  the 
training  questionnaire  are  available  in  appendix  C. 

3.3  Building  Reconnaissance  Course  Results 
3.3.1  Robotic  Control  Results 

Table  3  and  figure  8  show  the  mean  times  to  complete  the  course  using  each  of  the  operation 
modalities.  A  repeated  measures  ANOVA  showed  that  there  was  a  significant  difference  among 
the  means,  F(2,58)  =  17.4,  pcO.OOl,  >/  2P  =  0.383.  Follow-on  paired-sample  t-test  comparisons 
were  conducted  using  Holm’s  sequential  Bonferroni  correction  for  family-wise  error  rate.  As 
shown  in  table  4,  the  average  course  completion  time  in  the  teleoperation  condition  was 
significantly  slower  than  average  times  for  the  semi-autonomous  and  autonomous  conditions. 
The  slower  times  demonstrated  during  the  teleoperation  trials  can  be  attributed  to  the 
requirement  to  manually  map  the  buildings  as  well  as  an  increased  driving  error  rate.  The 
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average  course  completion  times  in  the  semi-autonomous  and  autonomous  conditions 
approached  significance  (p  =  0.069)  with  the  semi-autonomous  condition  being  slower  than  the 
autonomous  condition. 


Table  3.  Mean  course  completion  times  (min:sec). 


Condition 

Mean 

SD 

Teleoperation 

16:01 

3:54 

Semi-autonomous 

12:40 

4:06 

Autonomous 

11:21 

3:09 

Figure  8.  Mean  course  completion  times  (s)  with  95%  confidence  intervals. 
Table  4.  Follow-on  paired  comparisons,  course  completion  times. 


Pair 

t 

df 

Obtained  p 

Required  p 

Tele  vs.  Semi 

3.77 

28 

o.oor 

0.025 

Tele  vs.  Auto 

5.75 

29 

<o.oor 

0.0167 

Semi  vs.  Auto 

1.89 

28 

0.069 

0.05 

ap<0.05,  2-tailed 
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Table  5  and  figure  9  show  the  mean  number  of  driving  errors  with  the  three  operation  modalities. 
A  repeated  measures  ANOVA  indicated  that  there  was  a  significant  difference  among  the  means, 
F(2,58)  =  7.24,  p  =  0.002,  i]  2P  =  0.200.  Follow-on  paired  comparisons,  summarized  below  in 
table  6,  indicated  that  there  were  significantly  more  driving  errors  in  the  teleoperation  condition 
than  in  the  semi-autonomous  and  autonomous  control  conditions. 

Table  5.  Mean  driving  errors. 


Condition 

Mean 

SD 

Teleoperation 

3.67 

3.33 

Semi-autonomous 

1.33 

2.25 

Autonomous 

1.90 

2.32 

Figure  9.  Mean  driving  errors  with  95%  confidence  intervals. 
Table  6.  Follow-on  paired  comparisons,  driving  errors. 


Pair 

t 

df 

obtained  p 

required  p 

Tele  vs.  Semi 

4.14 

29 

<o.oor 

0.0167 

Tele  vs.  Auto 

2.52 

29 

0.017a 

0.025 

Semi  vs.  Auto 

-0.88 

29 

0.388 

0.05 

dp<0.05,  2-tailed 
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Based  on  the  post  iteration  questionnaire  results,  Soldiers  rated  the  task  of  mapping  the  building 
during  teleoperation  trials  as  the  most  difficult  task  of  the  reconnaissance.  In  the  autonomous 
and  semi-autonomous  conditions,  an  accurate  map  of  the  building  was  drawn  on  the  display  as 
the  robot  moved  through  the  building.  In  the  teleoperation  condition  the  participants  were  given 
lined  graph  paper  and  asked  to  draw  a  diagram  of  the  building,  complete  with  doors  and  items 
found  in  the  building.  Because  the  quality  of  some  of  the  drawings  was  so  poor,  it  was  difficult 
to  interpret  the  drawings.  We  conducted  a  reliability  check  by  having  two  independent  raters 
code  a  random  sample  of  fifteen  diagrams  for  the  number  of  doors,  the  number  of  doors  in  the 
correct  location,  and  the  total  area  (ft')  of  the  building.  The  reliability  coefficients  are  shown  in 
table  7.  Although  all  three  correlation  coefficients  were  statistically  significant,  there  was 
substantial  disagreement  between  the  two  raters  in  regard  to  the  number  and  location  of  doors  in 
the  diagrams. 


Table  7.  Inter-rater  reliabilities,  teleoperation  maps. 


Measure 

r 

P 

No.  doors 

0.65 

0.009 

No.  doors  correct  wall 

0.74 

0.002 

Area 

0.92 

0.000 

The  summary  statistics  from  the  map  drawing  ratings  are  shown  in  table  8.  Although  a  few 
participants  correctly  identified  all  of  the  doors  in  the  correct  location,  on  average  only  about  2/3 
of  the  doors  were  correctly  identified  and  drawn  in  the  correct  location.  All  of  the  participants 
underestimated  the  area  of  the  building,  most  by  a  substantial  amount.  One-sample  /-tests  were 
used  to  compare  each  mean  against  a  value  of  100%  that  would  be  obtained  using  the  maps 
automatically  generated  in  the  semi-autonomous  and  autonomous  driving  conditions.  Each  of 
the  means  was  significantly  lower  than  100%. 

Table  8.  Summary  statistics,  teleoperation  maps. 


Measure 

Mean 

SD 

t 

df 

P 

%  doors  identified 

70% 

25% 

4.75 

14 

<0.001 

%  correct  door  location 

67% 

27% 

4.70 

14 

<0.001 

Area 

40% 

20% 

11.83 

14 

<0.001 

3.3.2  Situation  Awareness  Results 

Table  9  shows  the  proportion  of  items  detected  (perception)  in  the  three  buildings  for  each 
robotic  control  condition.  A  repeated  measure  ANOVA  indicated  that  there  was  no  significant 
difference  among  the  means,  F<  1 .00. 


15 


Table  9.  Mean  proportion  of  items  detected. 


Condition 

Mean 

SD 

Teleoperation 

74% 

14% 

Semi-autonomous 

75% 

16% 

Autonomous 

72% 

11% 

3.3.3  Secondary  Task  Results 

During  the  building  reconnaissance  trials,  the  participants  were  periodically  asked  questions 
regarding  details  contained  in  the  operation  order,  or  they  were  asked  to  identify  specific  types  of 
grenades.  Soldiers  stated  that  performing  the  secondary  task  during  the  teleoperation  trials  was 
difficult  because  they  had  to  concentrate  more  of  their  attention  on  driving  and  manual  mapping 
tasks.  Table  10  and  figure  10  show  the  proportion  of  correct  responses  to  these  secondary  task 
questions.  A  repeated  measures  ANOVA  showed  that  there  was  a  significant  difference  among 
the  means,  F(2,58)  =  3.79,  p  =  0.028,  i]2p  =  0.115.  Follow-on  paired  comparisons,  shown  in 
table  1 1 ,  indicate  that  secondary  task  performance  was  significantly  better  in  the  autonomous 
condition  relative  to  the  teleoperation  condition. 

Table  10.  Mean  %  correct  secondary  activity. 


Condition 

Mean 

SD 

Teleoperation 

41% 

17% 

Semi-autonomous 

49% 

25% 

Autonomous 

55% 

25% 

16 


Figure  10.  Mean  %  correct  secondary  task  with  95%  confidence  intervals 


Table  11.  Follow-on  paired  comparisons,  secondary  task. 


Pair 

t 

df 

Obtained  p 

Required  p 

Tele  vs.  Semi 

-1.51 

29 

0.143 

0.025 

Tele  vs.  Auto 

-2.78 

29 

0.009a 

0.0167 

Semi  vs.  Auto 

-1.23 

29 

0.227 

0.05 

ap  <0.05,  2-tailed 

3.4  NASA-TLX  Results 

Table  12  shows  the  weighted  means  on  the  NASA-TLX  scales  and  the  total  workload  means. 
The  means  for  the  three  conditions  are  illustrated  in  figure  11.  Repeated  measures  ANOVAs  on 
the  scales  and  total  workload  are  summarized  in  table  13.  Follow-on  paired  comparisons  are 
shown  in  table  14.  There  were  no  significant  differences  across  conditions  on  the  Physical  and 
Performance  scales.  On  all  the  other  scales,  and  on  total  workload,  the  teleoperation  condition 
tended  to  have  the  highest  workload  means  and  the  autonomous  condition  had  the  lowest  means. 
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Table  12.  NASA-TLX  means. 


Condition 

Mental 

Physical 

Temporal 

Performance 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Teleoperation 

14.13 

8.62 

1.01 

2.38 

10.94 

7.99 

8.58 

4.88 

Semi-autonomous 

12.54 

8.75 

0.76 

2.26 

8.54 

6.25 

8.00 

4.91 

Autonomous 

10.10 

8.72 

0.87 

2.04 

6.54 

4.71 

8.07 

5.45 

Condition 

Effort 

Frustration 

Total  Workload 

— 

— 

Mean 

SD 

Mean 

SD 

Mean 

SD 

— 

— 

Teleoperation 

12.58 

8.02 

9.30 

10.79 

55.37 

22.39 

— 

— 

Semi-autonomous 

8.49 

7.14 

7.46 

9.18 

45.79 

19.70 

— 

— 

Autonomous 

7.29 

5.97 

3.93 

6.13 

37.82 

16.81 

— 

— 

Table  13.  ANOVA  summary,  NASA-TLX  means. 


Scale 

F 

df 

P 

>1 2 p 

Mental 

3.44 

2,58 

0.039“ 

0.106 

Physical 

<1 

2,58 

0.829 

0.006 

Temporal 

4.32 

2,58 

0.018“ 

0.130 

Performance 

<1 

2,58 

0.858 

0.005 

Effort 

9.14 

2,58 

<0.001  “ 

0.24 

Frustration 

4.31 

2,58 

0.018“ 

0.129 

Total  workload 

9.01 

2,56 

<0.001“ 

0.243 

“p  <0.05 
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Figure  11.  Mean  for  the  three  conditions  on  NASA-TLX. 
Table  14.  Follow-on  paired  comparisons,  NASA-TLX. 


Scale 

Pair 

t 

df 

Obtained  p 

Required  p 

Mental 

Tele  vs.  Semi 

0.93 

29 

0.361 

0.05 

Tele  vs.  Auto 

2.28 

29 

0.030 

0.025 

Semi  vs.  Auto 

2.30 

29 

0.029 

0.0167 

Temporal 

Tele  vs.  Semi 

1.47 

29 

0.152 

0.05 

Tele  vs.  Auto 

2.85 

29 

0.008a 

0.0167 

Semi  vs.  Auto 

1.54 

29 

0.135 

0.025 

Effort 

Tele  vs.  Semi 

3.28 

29 

0.003a 

0.025 

Tele  vs.  Auto 

3.82 

29 

0.001a 

0.0167 

Semi  vs.  Auto 

0.95 

29 

0.348 

0.05 

Frustration 

Tele  vs.  Semi 

0.98 

29 

0.337 

0.05 

Tele  vs.  Auto 

2.64 

29 

0.013a 

0.0167 

Semi  vs.  Auto 

2.17 

29 

0.038 

0.025 

Total  Workload 

Tele  vs.  Semi 

2.10 

29 

0.045a 

0.05 

Tele  vs.  Auto 

3.67 

28 

0.001a 

0.0167 

Semi  vs.  Auto 

2.80 

28 

0.009a 

0.025 

ap  <0.05,  2-tailed 
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3.5  Questionnaire  Results 


Soldiers  were  asked  to  indicate  their  first,  second,  and  third  choice  of  the  level  of  autonomy  they 
preferred  for  controlling  the  robot  while  performing  reconnaissance  tasks.  Table  15  shows  the 
results  of  their  ratings;  the  higher  the  mean,  the  greater  the  preference. 

Table  15.  Level  of  autonomy  preferences  for  performing  reconnaissance  tasks. 


Mean  Rating 

Teleoperation 

Semiautonomous 

Operation 

Autonomous 

Operation 

Route  planning 

1.62 

2.04 

2.35 

Generating  goals 

1.88 

1.96 

2.20 

Driving  a  straight  route 

1.96 

2.15 

1.85 

Avoiding  obstacles 

1.81 

2.04 

2.15 

Looking  for  objects  of  interest 

2.38 

1.96 

1.81 

Overall  building  reconnaissance 

1.73 

1.96 

2.31 

For  the  most  part,  the  Soldiers’  choices  for  the  level  of  autonomy  for  reconnaissance  tasks 
indicated  that  the  more  autonomy  the  better.  Their  choices  for  such  tasks  as  overall  building 
reconnaissance,  avoiding  obstacles,  generating  goals,  and  route  planning  showed  their 
preferences  for  more  autonomy.  For  driving  a  straight  route,  the  Soldiers  preferred 
semiautonomous  operations  because  they  felt  that  they  could  drive  the  robot  faster  than  it 
traversed  in  the  autonomous  mode.  They  preferred  less  autonomy  when  looking  for  objects  of 
interest  because  they  could  spend  the  amount  of  time  they  needed  and  go  closer  to  the  object  in 
order  to  identify  it. 

The  Soldiers  were  very  positive  about  the  control  system  because  they  found  it  to  be  easy  to 
learn  and  to  use.  Many  found  it  to  be  familiar  because  they  had  spent  a  lot  of  time  using  Xboxes. 
There  were  several  complaints  that  the  camera  on  the  robot  did  not  have  adequate  resolution  and 
that  they  needed  to  be  able  to  pan,  tilt,  and  zoom  the  camera  in  order  to  identify  objects  and  take 
pictures  more  easily.  They  also  experienced  some  difficulty  seeing  the  display  because  the  room 
in  which  they  were  operating  the  robot  was  very  bright  and  glare  was  a  problem.  A  few  Soldiers 
thought  that  a  color  display  would  assist  them  in  identifying  objects.  Soldiers  also  complained 
about  the  latency  in  the  system  which  especially  impacted  teleoperation. 

When  asked  to  choose  if  they  preferred  the  autonomous  or  semi-autonomous  modes,  the  Soldiers 
were  closely  split,  with  16  preferring  the  autonomous  and  14  preferring  the  semi-autonomous. 
Those  preferring  the  autonomous  mode  did  so  because  it  freed  them  to  do  other  tasks  and  it 
seemed  to  work  smoother.  Those  that  preferred  the  semi-autonomous  mode  did  so  because  they 
could  make  the  robot  move  faster  and  they  were  able  to  provide  input.  Some  Soldiers  did  not 
like  turning  over  control  to  the  robot  at  all.  Two  of  them  wanted  total  control  at  all  times.  One 
Soldier  commented  that  the  use  of  both  the  semi-autonomous  and  autonomous  modes  allowed 
him  to  shift  his  focus  to  the  secondary  task  while  maintaining  progress  toward  his  primary 
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objective.  Due  to  the  small  size  of  the  area  being  reconnoitered  and  the  number  of  objects  in 
each  room,  Soldiers  constantly  had  to  take  manual  control  and  teleoperate  the  robot  into  position 
to  take  snapshots  of  the  items  during  the  autonomous  and  semi-autonomous  trials. 

Drawing  an  accurate  map  in  the  teloperation  condition  was  especially  difficult.  The  Soldiers 
were  extremely  positive  about  the  ability  of  the  robot  to  build  maps  of  the  room  and  used  the 
map  frequently.  Soldiers  found  the  mapping  capability  in  the  autonomous  and  semi-autonomous 
modes  particularly  useful  for  gauging  their  progress  while  maneuvering  through  the  building. 


4.  Discussion  and  Recommendations 


Total  vehicle  reconnaissance  times  and  driving  errors  were  significantly  better  when  the  robot 
was  in  the  autonomous  and  semi  autonomous  modes  than  when  it  was  in  the  teleoperation  mode. 
Scribner  and  Dahn  (2008)  found  very  little  difference  in  driving  times  between  teleoperation  and 
their  version  of  semi-autonomy  called  the  biasing  mode  (the  robot  semi-autonomously  follows  a 
previously  recorded  GPS  path  with  the  operator  providing  course  corrections  for  GPS  path  error 
and  obstacle  detection  and  avoidance).  However,  differences  in  the  levels  of  autonomy  of  the 
robots  in  the  two  experiments  can  explain  the  different  results.  The  biasing  condition  was 
different  than  the  semi-autonomous  mode  of  this  experiment  because  the  semi-autonomous  mode 
included  autonomous  obstacle  avoidance  and  the  biasing  mode  required  the  operator  to  perform 
obstacle  avoidance.  The  biasing  mode  also  did  not  have  nearly  the  autonomy  found  in  the 
autonomous  mode  of  this  experiment.  The  latency  (the  lag  between  a  control  input  and  the 
response  of  the  robot)  complained  about  by  the  operators  may  also  have  had  an  impact  on  the 
results  as  it  could  have  had  more  of  a  negative  impact  on  teleoperation  times  and  driving  errors 
than  on  operation  in  the  semi-autonomous  and  autonomous  modes.  Lane  et  al.,  (2002)  indicated 
that  system  latency  can  actually  change  robotic  operator  control  strategy  from  continuous 
command  to  “move  and  wait”  which  adversely  impacts  reconnaissance  times.  Over-actuation 
which  is  common  when  latency  is  unpredictable  (Malcolm  and  Lim,  2003)  can  also  create 
driving  errors. 

The  consensus  of  the  Soldiers  was  that  mapping  the  room,  driving  the  robot,  and  answering  the 
questions  in  the  teleoperation  mode  created  task  demands  that  were  too  difficult.  On  the  NASA- 
TLX,  the  Soldiers  rated  teleoperation  as  creating  a  higher  cognitive  and  overall  work  load,  higher 
stress  (frustration),  more  effort,  and  higher  time  pressure.  While  Dixon  and  Wickens  (2003) 
suggested  that  automation  would  relieve  cognitive  overload,  offloading  some  of  the  tasks  by  use 
of  the  autonomous  modes  relieved  more  than  just  the  cognitive  load  and  made  the  performance 
of  the  tasks  much  more  manageable.  This  data  is  supported  by  Schipani  (2003)  who  stated  that 
workload  is  increased  for  higher  levels  of  operator  involvement.  Map  accuracy  in  the 
teleoperation  mode  was  much  poorer  than  in  the  autonomous  and  semi-autonomous  modes.  It 
was  clear  that  the  operators’  mental  models  of  the  environment,  based  upon  viewing  it  through  a 
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robotic  driving  camera,  were  fairly  inaccurate.  This  is  supported  by  the  findings  of  Fong  et  al. 
(2003)  who  indicated  that  operators  using  teleoperation  have  difficulty  building  mental  models 
of  remote  environments. 

Results  in  the  literature  concerning  SA  and  targets  (objects)  identified  while  the  robot  is  moving 
are  mixed  concerning  whether  higher  levels  of  autonomy  result  in  better  SA  or  vice  versa.  Chen 
and  Joyner  (2006)  found  that  participants  detected  fewer  targets  when  their  robot  was  operating 
in  the  semi-autonomous  mode  rather  than  the  teleoperation  mode.  However,  automation  seemed 
to  benefit  UAV  pilots’  target  detection  performance  in  the  Dixon  et  al.,  (2003)  study.  Our  results 
were  different  from  both  of  the  previous  studies  in  that  no  significant  differences  were  found 
between  the  levels  of  automation.  However,  if  SA  is  more  broadly  defined  in  the  Chen  and 
Joyner  (2006)  study  to  include  the  gunner  task  and  the  communication  task,  participants  did  not 
have  better  SA  in  the  teleoperation  condition.  The  complex  nature  of  SA  and  its  definition 
makes  it  important  to  specify  the  definition  of  SA  when  making  comparisons  between  studies. 


5.  Conclusions 


Findings  from  this  experiment  indicate  that  as  levels  of  autonomy  increase,  workload, 
reconnaissance  times,  and  driving  errors  decrease  and  accuracy  of  mental  maps  increase.  No 
differences  were  found  in  SA  and  target  identification  between  levels  of  autonomy.  Future 
research  should  be  conducted  to  further  define  the  areas  in  which  automation  can  improve 
performance  and  decrease  workload. 
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Appendix  A.  Operations  Order  (Engineering  Reconnaissance 

[FM  5-170],  1998) 


This  appendix  appears  in  its  original  form  without  editorial  change. 
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Copy  1_of  10  copies 
HQ,  99th  Engineer  Battalion 
NK111111 
080500  SEP  09 


OPERATION  ORDER  10-11 


References: 

1st  Bde  OPORD  10-23 

Map  sheet  VI 07 

Time  Zone  Used  Throughout  The  Order:  Local 


Task  Organization: 


A/99  En  Bn 

B/99  En  Bn 

1/C/99  En  Bn 

Bn  control 

C/99  En  Bn  (-) 

Recon  Team  1 

1.  SITUATION, 
a.  Enemy  Forces. 

(1)  Terrain  and  Weather. 

(a)  Observation  is  generally  limited  along  the  valley  floor  due  to  the  terrain's  undulating 
nature.  Multiple  intervisibility  lines,  generally  running  north  to  south  and  spaced  between 
500  to  1 ,000  meters,  will  hamper  observation.  Movement  to  the  higher  elevations  along 
either  the  north  or  south  wall  will  obviously  improve  observation.  Winds  (expected  to 
exceed  20  knots  until  at  least  1 11200  SEP  09)  will  lift  sand  from  the  desert  floor  and 
hamper  observation.  Observation  at  night  will  be  extremely  limited  due  to  the  light  data 
for  the  next  72  hours.  Note  that  on  8  SEP  09  the  moon  sets  before  the  sun  and  on  9 
through  1 1  SEP,  the  moon  sets  soon  after  the  sun;  therefore,  night-vision  goggles 
(NVGs)  will  provide  limited  capabilities  for  the  next  72  hours  and  make  observation, 
movement,  and  the  acquisition  of  OBSTINTEL  more  difficult. 
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Date 

BMNT 

SR 

SS 

EENT 

MR 

MS 

Start 

NVG 

Stop 

NVG 

%  Ilium 

8  SEP 

0555 

0654 

1701 

1800 

0550 

1633 

***** 

***** 

0% 

9  SEP 

0555 

0653 

1702 

1801 

0643 

1743 

***** 

***** 

0% 

10 

SEP 

0555 

0653 

1702 

1802 

0731 

1853 

***** 

***** 

4% 

11 

SEP 

0554 

0653 

1704 

1803 

0813 

2002 

***** 

***** 

9% 

(b)  The  only  cover  from  both  direct  and  indirect  fires  is  provided  by  the  undulating 
terrain  previously  mentioned.  Concealment  during  movement  can  be  enhanced  by 
traveling  parallel  to  the  intervisibility  lines  when  available.  The  dusty  and  windy 
conditions  may  make  mounted  movement  less  detectable  by  the  enemy. 

(c)  The  pipeline  running  parallel  to  the  LD  along  the  30  easting  is  the  only  existing 
obstacle  in  the  AO.  Crossing  points  for  this  pipeline  have  been  identified  at  NK  302215 
and  NK  295090. 

(d)  The  terrain  in  the  vicinity  of  the  templated  obstacle  system  is  believed  to  be 
unsuitable  for  minefield  reduction  by  MCBs  because  of  the  undulating  terrain  and  the 
soil  composition. 

(2)  Enemy  Situation. 

(a)  The  133d  motorized  rifle  battalion  (MRB)  is  currently  preparing  defenses  along  the 
47  easting.  This  unit's  expected  strength  is  estimated  to  be  12  T-80s,  32  BMP-1  s,  3  AT- 
5s,  and  1  dismounted  infantry  company.  The  133d  MRB  began  preparing  its  defenses 
071500  SEP  and  are  not  expected  to  complete  its  counter  mobility  and  survivability 
effort  before  091600  SEP.  The  133d  MRB  is  expected  to  have  a  company-size 
combined-arms  reserve  at  a  strength  of  three  T-80s  and  eight  BMP-1  s. 

(b)  As  of  080100  SEP,  three  enemy  MRCs  have  been  located  and  are  depicted  on  the 
SITEMP.  The  expected  positioning  of  the  subordinate  MRPs  is  also  templated  as  well 
as  the  anticipated  combat  security  observation  post  (CSOP)  and  artillery  positions. 
Expect  to  come  within  direct-fire  range  of  the  CSOPs  when  crossing  the  42  easting  and 
the  main  defenses  when  crossing  the  44  easting.  Enemy  artillery  is  expected  to  be  in 
position  not  later  than  (NLT)  081600  SEP;  expect  to  come  within  indirect-fire  range 
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when  crossing  the  25  easting.  However,  the  enemy  will  rarely  use  indirect  fires  against 
recon  forces.  Expect  the  enemy  to  use  its  rotary-wing  assets  in  its  attempt  to  try  to 
locate  and  destroy  recon  forces.  The  enemy  is  not  expected  to  be  supported  by  fixed- 
wing  aircraft.  Although  the  enemy  has  the  capability  to  employ  chemical  weapons,  it  has 
chosen  not  to  do  so  thus  far  in  the  campaign.  However,  if  the  enemy  does  employ 
chemicals,  we  expect  them  to  emplace  a  persistent  chemical  agent  at  center  of  mass 
NK  41 0280. 

The  enemy  is  not  nuclear  capable.  Expect  the  enemy  to  use  dismounted  strong  points 
to  tie  its  obstacles  into  the  restricted  terrain  at  vicinities  NK  450210  and  NK  440100. 
These  dismounted  forces  will  be  supplied  with  AT-5s  to  assist  in  their  mission  of 
preventing  the  obstacle  system  from  being  reduced  along  the  walls  of  the  valley. 
Additionally,  the  enemy  will  use  dismounted  patrols  to  protect  all  minefields. 

(c)  The  templated  obstacle  system  is  included  on  the  SITEMP.  No  confirmed  obstacle 
locations  have  been  obtained  as  of  080100  SEP.  We  expect  the  enemy  to  continue  to 
lay  its  minefields  similar  to  the  method  used  throughout  the  campaign.  We  expect  the 
enemy  to  mechanically  lay  its  minefields  and  expect  each  minefield  to  be  comprised  of 
SB-MV  mines  and  be  200  to  300  meters  long  and  60  to  120  meters  deep.  The  mine 
spacing  has  consistently  been  4.5  meters  and  the  depths  of  the  mines  have  been  up  to 
9  inches.  NOTE:  The  SB-MV  is  magnetic-influence  initiated  and  must  be  detected 
by  probing.  Operating  hand-held  mine  detectors  may  detonate  the  mine.  If  mines 
are  surface  laid,  it  is  probably  due  to  the  soil  conditions  and  indicates  the  probable 
success  of  using  MCBs.  The  enemy  has  routinely  used  a  single-strand  of  concertina 
fence  on  the  enemy  side  of  the  minefield  as  a  frat  fence.  The  enemy  is  expected  to 
emplace  a  total  of  15  minefields  in  its  defense. 

b.  Friendly  Forces. 

(1)  Higher. 

(a)  1st  Brigade  plans  to  conduct  a  brigade  breaching  operation  and  penetrate  the 
northern  MRP  of  the  northern  MRC  as  shown  on  the  SITEMP.  Other  brigade  recon 
assets  include  two  COLTs  and  one  chemical  recon  vehicle.  The  planned  locations  for 
each  of  these  assets  are  shown  on  the  maneuver  graphics^. 

(b)  Engineer  recon  team  1  is  attempting  to  answer  the  brigade  commander's  PIR  for 
location,  composition,  and  orientation  of  the  enemy's  obstacles. 

(c)  If  bypasses  of  the  enemy  obstacles  can  be  located,  the  brigade  commander  would 
prefer  to  bypass  the  obstacles  as  close  to  the  north  wall  as  possible. 

(2)  Lower.  Do  not  expect  TF  recon  assets  to  cross  the  LD  before  EENT  on  9  SEP  09. 
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2.  MISSION. 


The  99th  Engineer  Battalion  conducts  an  area  recon  of  NAI  301  NLT  082000  SEP  09  to 
facilitate  the  brigade's  attack  at  110500  SEP  09. 

3.  EXECUTION. 

Intent.  The  purpose  of  this  mission  is  to  identify  enemy  obstacles  within  NAI  301  to 
confirm  or  deny  the  enemy's  COA  and  facilitate  breaching  operations.  The  end  state  is 
the  identification  of  enemy  obstacles  in  NAI  301  NLT  100500  SEP  09  and  recon  team  1 
in  position  at  checkpoint  (CP)  15  ready  to  link  up  and  guide  the  breach  force  to  the 
obstacle  location  NLT  110001  SEP  09. 

a.  Concept  of  Operation. 

The  battalion  conducts  an  obstacle-oriented  area  recon.  Recon  team  1  will  cross  the  LD 
NLT  082000  SEP.  The  brigade  will  have  at  least  two  batteries  ready  to  provide  indirect 
fires  out  to  PL  Celtics  throughout  the  recon  effort,  and  the  attack  helicopter  battalion 
(AHB)  will  support  casualty  evacuation.  Team  1  will  cross  the  LD  about  24  hours  before 
the  TF  scouts  in  an  attempt  to  observe  and  report  the  enemy  emplaced  obstacles  and 
signs  of  recent  enemy  activity  while  the  TF  is  still  planning  its  R  &  S  effort.  The  recon 
team  will  link  up  with  TF  1-23  scouts  (who  will  provide  security)  before  conducting 
obstacle  recon.  Recon  team  1  will  complete  its  area  recon  of  NAI  301  NLT  100500  SEP 
to  facilitate  mounted  rehearsals  by  the  brigade  during  daylight  hours  on  1 1  SEP  09. 
Recon  team  1  will  continue  to  observe  NAI  301  until  101700  SEP  and  report  any  further 
engineer  activity.  At  101800  SEP  09,  recon  team  1  will  move  to  CP  15  and  be  in 
position  NLT  110001  SEP  09,  prepared  to  link  up  and  guide  the  breach  force  to  the 
obstacle  location. 

b.  Tasks  to  Subordinate  Units. 

(1)  Battalion's  TOC.  The  battalion's  TOC  will— 

(a)  Provide  liaison  personnel  to  co-locate  with  the  recon  team  until  they  cross  the  LD 
and  ensure  that  liaison  personnel  obtain  a  copy  of  the  recon  team's  maneuver  graphics. 

(b)  Coordinate  the  recon  team's  indirect  fire  plan  with  the  FSO  and  confirm  targets  with 
the  team  leader  once  they  are  coordinated. 

(2)  Battalion  S4.  The  battalion  S4  will  obtain  the  current  logistical  status  of  recon  team 
1.  Fie  will  ensure  that  unit  basic  load  (UBL)  levels  are  reestablished  NLT  081200  SEP 
09  and  report  to  the  TOC  upon  completion. 

(3)  A/99  En  Bn.  A/99  En  Bn  will  conduct  liaison  activities  between  recon  team  1  and  TF 
1-23  according  to  the  battalion's  TACSOP. 
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(4)  Recon  team  1 .  Recon  team  1  will— 

(a)  Report  the  current  logistical  status  to  the  S4  NLT  080800  SEP  09. 

(b)  Backbrief  the  plan  to  the  battalion  commander  via  FM  radio  at  081300  SEP  09. 

(c)  Provide  TF  1-23  the  team's  graphics,  via  the  A/99  En  Bn's  TOC  before  crossing  the 
LD. 

(d)  Forward  requested  indirect-fire  targets  to  the  battalion's  TOC  NLT  081600  SEP  09. 

(e)  Coordinate  link  up  with  the  TF  1-23  scouts  for  security  during  obstacle  recon. 

(f)  Conduct  an  area  recon  of  NAI  301  NLT  082000  SEP  09  to  verify  the  composition  of 
obstacles  within  the  NAI. 

c.  Coordinating  Instructions. 

(1)  Task  organization  is  effective  upon  receipt  of  this  order. 

(2)  All  units  will  participate  in  the  intelligence  updates  to  occur  at  0800  and  2000  each 
day. 

(3)  The  LOA  for  recon  assets  is  PL  Celtics. 

4.  SERVICE  SUPPORT. 

a.  Support  Concept. 

(1)  The  recon  team  will  cross  the  LD  fully  uploaded  according  to  the  battalion's 
TACSOP.  These  supplies  will  come  from  the  engineer  battalion.  This  basic  load  is 
expected  to  sustain  the  team  throughout  the  mission. 

(2)  Emergency  resupply  will  be  coordinated  through  the  engineer  battalion's  TOC  and 
delivered  by  aviation  assets.  Backup  resupply  will  be  through  TF  1-23. 

b.  Medical  Evacuation  and  Hospitalization.  The  primary  means  of  MEDEVAC  is  by  air 
(requested  through  the  battalion's  TOC);  backup  is  by  ground  evacuation  (performed  by 
TF  1-23). 

c.  Personnel  Support.  EPWs  will  be  turned  over  to  TF  1-23  for  evacuation  to  the  rear. 

5.  COMMAND  AND  SIGNAL. 

a.  Command.  The  chain  of  command  is  the  commander,  the  XO,  the  S3,  and  the 
commander  of  C  Company. 
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b.  Signal. 

(1)  All  traffic  from  recon  team  1  to  the  battalion's  TOC  will  be  over  MSRT  (primary)  or 
the  battalion's  command  net  (alternate). 

(2)  The  recon  team's  current  location  will  be  sent  by  the  battalion's  TOC  to  TF  1-23. 

(3)  OBSTINTEL  will  be  reported  according  to  the  TACSOP. 

ACKNOWLEDGE: 

PATTON 

LTC 
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Appendix  B.  Demographics 


This  appendix  appears  in  its  original  form  without  editorial  change. 
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SAMPLE  SIZE  =  30 


MOS 

RANK 

DUTY  POSITION 

09S-  10 

68D-  1 

E4-6 

Air  Aslt  Instructor  -  1 

Pathfinder  Instructor  - 1 

11B-8 

68W  -  1 

E5-  17 

Crew  Chief  -  1 

Ranger  Instructor  -  1 

11C-  1 

88M-2 

E6-2 

Driver  -  1 

Support  - 1 

12B-  1 

OCS-3 

OCS-4 

Instructor  -  1 

Team  leader  -  4 

15T-  1 

NR-  1 

NR-  1 

Medic  -  1 

Training  -  1 

31 B-  1 

OCS-7 

Combat  Engr  -  1 

NR -9 

AGE 


28  years  (range  21-47) 

1 .  How  long  have  you  served  in  the  military?  65  months  (mean) 

2.  How  long  have  you  had  an  infantry-related  job?  72  months  (mean) 

3.  How  long  have  you  been  a  fire  team  leader?  21  months  (mean) 

4.  How  long  have  you  been  a  squad  leader?  _44_  months  (mean) 

5.  How  long  have  you  been  deployed  overseas?  _21_  months  (mean) 

6.  How  long  have  you  been  deployed  in  a  combat  area?  J_5_  months  (mean) 

7.  With  which  hand  do  you  most  often  write?  25  Right  _5_  Left 

8.  With  which  hand  do  you  most  often  fire  a  weapon?  27  Right  _3_  Left 

9.  Do  you  wear  prescription  lenses?  _4_  Yes  26  No 

10.  If  yes,  which  do  you  wear  most  often?  _2_  Glasses  _2_  Contacts 

1 1 .  Which  is  your  dominant  eye?  26  Right  4  Left 

12.  Do  you  have  any  vision  related  problem?  _2_  Yes  27  No  1  NR 
If  so,  what?  Red/green  color  blind  (1),  farsighted  (1) 

13.  Have  you  ever  used  a  robotic  system?  J_  Yes  23  No  6  NR 
If  so,  what  type?  Davinci  (1) 

14.  Please  describe  the  conditions  under  which  you  used  the  robotic  system. 

Surgical  (1) 

15.  Using  the  scale  below,  please  rate  your  skill  level  for  each  of  the  following  activities. 
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None  Beginner  Intermediate  Expert 
12  3  4 


ACTIVITY 

MEAN  RESPONSE 

Operating  ground  unmanned  vehicles 

1.13 

Operating  aerial  vehicles 

1.09 

Target  detection  and  identification 

1.55 

Playing  commercial  video  games 

2.65 

Training  with  Army  video  simulations 

2.17 
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Intentionally  left  blank. 
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Appendix  C.  Training  Questionnaire 


This  appendix  appears  in  its  original  form  without  editorial  change. 
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SAMPLE  SIZE  =  30 


1 .  Using  the  scale  below,  please  rate  the  training  you  received  in  the  following  areas. 

1  2  3  4  5  6  7 

Extremely  bad  Very  bad  Bad  Neutral  Good  Very  good  Extremely  good 


MEAN  RESPONSE 

a.  Introductory  robotic  training 

-Completeness  of  introductory  training 

5.90 

-Comprehension  of  the  overall  concept  of  the  robot 

5.90 

b.  Building  reconnaissance  training 

(1)  Robotic  driving 

-How  to  drive  the  robot  (teleoperation) 

5.73 

-How  to  drive  the  robot  (semiautonomous) 

5.70 

-How  to  drive  the  robot  (autonomous) 

5.97 

(2)  Time  provided  to  practice  driving  the  robot  on  the  building 
reconnaissance  course 

-Teleoperation 

5.59 

-Semiautonomous 

5.55 

-Autonomous 

5.81 

(3)  How  to  complete  the  building  reconnaissance  course 
(independent  of  levels  of  autonomy) 

c.  Overall  evaluation  of  the  robotic  training  courses 

5.96 

No.  of 

Comments  Responses 

I  personally  was  challenged;  not  a  video  gamer.  1 

Instructions  were  great!  1 

b.  Building  Reconnaissance  Training 

I  liked  the  tele  mode  best.  1 

More  time  makes  you  better,  but  for  what  we  did  today,  it  was  enough.  1 

The  auto  was  the  easiest  level  because  it  gives  you  the  opportunity  to 
multitask  a  lot  easier  than  the  other  two  settings.  1 

More  distractions  perhaps.  1 

Attempting  to  draw  to  scale,  take  pictures,  and  maintain  awareness  may 
be  a  problem  in  the  tactical  field.  1 

Drawing/mapping  was  difficult.  1 

c.  Overall  Robotic  Training 

Extremely  fun  and  helpful  for  the  future  of  operation.  1 

Great  ideas,  tech,  instruction.  1 

d.  Speech  Training 

Would  free  up  more  time  and  frustration.  1 

Need  a  little  more  time  with  it.  I  felt  like  I  had  more  time  with  grenades 
and  the  robot  besides  the  speech  control.  1 

Not  good  at  all  in  my  case,  but  that  has  to  do  with  my  ability  in  speaking 
English.  1 
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2.  What  were  the  easiest  and  hardest  training  tasks  to  learn? 


No.  of 

Comments  Responses 


Easiest 

All  of  them.  1 

Training  on  all  was  performed  well.  1 

The  task  of  each  control.  1 

Control  of  robot  (driving,  steering,  moving,  maneuvering,  etc.).  12 

Use  of  controller.  1 

Using  controls  to  draw  the  map.  1 

Learning  how  the  remote  control  works,  and  what  buttons  do  what.  2 

Taking  pictures.  8 

Collect  intel.  2 

Photographing  and  identification.  1 

Identify  weapons  and  other  items.  1 

Labeling  pictures.  3 

Identifying  obstacles.  1 

Autonomous  was  the  easiest.  2 

Tele  and  Semi  were  the  easiest.  1 

Hardest 

None  of  them,  at  least  to  me.  1 

Doing  the  entire  job.  Better  to  just  be  in  the  room.  1 

Feeling  the  turn  radius  of  robot.  1 

Getting  robot  to  move  easier.  They  should  find  a  way  to  have  pivot 
points  instead  of  tracks  so  the  robot  eye  is  multidirectional  and 
independent  of  the  mobility  of  the  robot  itself.  1 

Robot  has  difficulty  with  full  autonomous  mode  in  close  areas.  1 

Prioritizing  the  video  with  the  map  for  situational  awareness.  1 

How  to  work  my  way  through  the  building.  I  didn’t  like  the  delay  using 
the  map  in  relation  to  my  controls  was  unclear.  1 

Mapping  the  building  by  hand.  4 

Drawing  the  map  during  building  reconnaissance.  1 

Answering  questions  while  driving  robot.  3 

Situational  awareness.  1 

Object  identification.  1 

Driving  around  obstacles.  1 

Speech  training/controls.  2 

Taking  pictures.  But  once  you  get  used  to  it,  it  becomes  very  easy.  1 

Labeling  the  pictures.  1 

Overall  functions  of  each  button.  2 

Using  the  joystick/control  pads.  1 

Understanding  that  you  have  “tunnel  vision”  with  the  robot.  1 

Tele  was  the  hardest.  2 

Autonomous  was  the  hardest.  1 
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3.  What  are  your  overall  comments  on  the  training? 


Auto  and  Semi  worked  great.  1 

Good. 

Good  learning  experience;  something  I’ve  never  seen  before. 

Fantastic  tool.  Can  see  how  this  will  save  more  lives  by  using  the  robot 
as  the  recon  tool.  1 

Interesting  and  fun.  3 

Easily  understood  what  to  do  during  these  exercise.  1 

Fairly  simple.  1 

Cadre  was  very  informative.  1 

I’m  glad  I  got  to  participate.  1 

Would  like  to  see  how  it  works  with  a  two-man  team  with  the  same 
evaluation.  1 

I  feel  everyone  was  very  knowledgeable  and  professional.  1 

Full  instructions;  good  review.  1 

I  felt  very  comfortable  operating  the  robot.  1 

Could  use  on  battlefield  in  urban  areas.  1 

I  would  like  to  do  it  in  a  real  mission.  1 

Could  use  more  time  to  practice  controlling  robot.  1 

Driving  the  robot  and  not  running  into  everything,  like  the  edge  of  a  door.  1 

Display  hard  to  see.  1 

The  use  of  the  directional  pad  would  have  been  great  to  have  known 
about  in  the  training  so  I  could  practice.  1 

The  tele  operation  and  map  drawing  was  extremely  difficult  to  do  both 
under  a  set  time.  1 
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Appendix  D.  Post  Iteration  Questionnaire 


This  appendix  appears  in  its  original  form  without  editorial  change. 
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SAMPLE  SIZE  =  30 


AUTONOMY  LEVEL: 

TELEOPERATION  (TELE).  SEMIAUTONOMOUS  (SEMI).  AUTONOMOUS  (AUTO) 

1 .  Using  the  scale  below,  please  rate  your  ability  to  perform  each  of  the  following  tasks 
based  on  your  experience  with  the  autonomy  level  that  you  just  used. 


1  2  3  4  5  6  7 

Extremely  Very  Difficult  Neutral  Easy  Very  Extremely 

difficult  Difficult  easy  easy 


MEAN  RESPONSE 

TELE 

SEMI 

AUTO 

a.  Move  the  robot  in  the  correct  direction 

5.23 

5.20 

6.00 

b.  Avoid  obstacles 

5.14 

5.50 

5.48 

c.  Avoid  pot  holes 

5.67 

5.50 

7.00 

d.  Assess  down  slopes  for  navigatability 

5.50 

5.67 

7.00 

e.  Assess  side  slopes  for  navigatability 

5.50 

5.67 

7.00 

f.  Identify  any  other  terrain  features  that  might  have  an 
adverse  effect  on  the  ability  of  the  robot  to  maneuver 
through  the  terrain 

6.13 

6.38 

6.00 

g.  Anticipate  whether  the  ground  clearance  of  the  vehicle  will 
allow  negotiation  of  rugged  terrain 

6.25 

6.80 

6.67 

h.  Anticipate  whether  the  turn  radius  of  the  vehicle  will  allow 
a  turn 

4.67 

4.65 

5.32 

i.  Identify  if  the  robot  is  on  the  correct  path 

5.38 

5.37 

5.72 

j.  Navigate  far  enough  ahead  to  plan  route  in  advance 

5.14 

5.21 

5.27 

k.  Navigate  well  enough  to  drive  at  slowest  speeds 

6.00 

6.17 

6.22 

1.  Navigate  well  enough  to  drive  at  medium  speeds 

5.75 

6.00 

6.05 

m.  Navigate  well  enough  to  drive  at  fastest  speeds 

5.32 

5.38 

5.74 

n.  Finish  the  course  quickly 

4.07 

4.75 

5.34 

o.  Ability  to  find  lEDs  and  other  objects  of  interest 

5.69 

5.86 

6.00 

p.  Ability  to  navigate  to  the  next  waypoint 

5.50 

5.68 

5.72 

q.  Ability  to  map  the  room  (building  reconnaissance  course 
only) 

3.75 

6.00 

6.21 

r.  Ability  to  take  pictures 

6.40 

6.29 

6.37 

s.  Ability  to  make  the  robot  understand  you 

5.88 

5.21 

5.65 

t.  Ability  to  maintain  situation  awareness 

4.37 

4.97 

5.38 

u.  Overall  ability  to  perform  this  reconnaissance  with  this 
level  of  autonomy 

4.35 

5.20 

5.86 
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No.  of 

Comments  Responses 

TELE 

Mapping  room  accurately  was  difficult.  1 

If  the  camera  on  the  robot  could  zoom  in  and  out,  it  would  help  out  a  lot.  1 

Camera  does  not  allow  any  view  other  than  forward  at  ground  level. 

Can’t  tell  how  close  it  is  to  objects.  Can’t  see  anything  that  could  be 
happening  behind  robot,  i.e.,  rear  camera.  1 

This  mode  is  difficult  as  one  person.  If  a  two-man  team  was  involved,  it 
would  be  easier  by  a  lot.  1 

SEMI 

Seems  the  easiest  and  most  effective.  1 

Good,  because  you  can  let  the  robot  move  on  its  own  and  stop  it  and 
take  control  if  you  wanted  to.  1 

Performing  functions  on  the  controller  became  easy  to  remember  after  a 
few  repetitions;  approx  10  minutes.  1 

The  directional  pad  wouldn’t  respond  well  (slow  or  not  at  all).  The  robot 
moves  extremely  slow  and  “thinks”  too  much  for  each  breach  of  a 
room.  It  was  faster  to  just  let  it  go  into  a  room  and  then  take  control.  1 

The  joystick  sensitivity  is  not  high  enough.  1 

In  my  opinion,  this  level  is  worthless  because  I  had  to  operate  it  more 
than  by  itself.  1 

I  prefer  the  autonomous  more.  1 

AUTO 

It  was  easy  for  me  to  answer  questions  and  look  up  info  in  this  mode.  1 

Easier  with  practice.  1 

Overall  felt  like  the  same  as  semi.  Seemed  more  efficient  to  continually 
take  over  rather  than  allow  robot  to  maneuver  on  its  own  the  whole  time 
due  to  number  of  items.  1 

Unable  to  label  some  objects  due  to  low  camera  resolution  or  perhaps 
light  levels.  1 


2.  Please  check  any  of  the  following  conditions  that  you  may  have  experienced  during  this 
trial. 


NUMBER ( 
RESPONSI 

DF 

ES 

TELE 

SEMI 

AUTO 

Eyestrain 

7 

7 

4 

Tunnel  vision 

2 

2 

1 

Headaches 

0 

1 

0 

Motion  sickness 

0 

0 

0 

Nausea 

0 

0 

0 

Disorientation 

2 

0 

0 

Dizziness 

0 

0 

0 
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Competition  between  eyes  for  vision  of  different 
scenes  at  which  they  are  looking 

2 

3 

1 

Any  other  problems? 

0 

1 

0 

Comments 

TELE 


BUILDING  RECONNAISSANCE 


Competition  between  eyes  when  trying  to  look  at  the  screen  and  draw  at 
the  same  time  on  your  map.  If  I  were  the  operator,  I  would  have  an 
assistant  to  map  it. 

Eyestrain  from  glare  on  the  computer  screen. 

Screen  should  be  bigger  or  be  in  room  with  less  light. 

SEMI 

Some  items  were  a  little  hard  to  make  out  with  the  camera  and  caused 
eyestrain. 

AUTO 

Training  is  easy. 

Some  items  were  a  little  hard  to  make  out  with  the  camera  and  caused 
eyestrain. 

I  would  be  able  to  ID  smaller  objectives  if  there  was  a  zoom  feature  on 
the  camera. 

Possible  color  vision. 

Larger  screens,  even  for  training  purpose. 


No.  of 
Responses 


1 

1 

1 


1 

1 

1 

1 

1 

1 


3.  Using  the  scale  below,  what  is  your  overall  rating  of  the  level  of  autonomy  you  used  on 
this  course. 

1  2  3  4  5  6  7 

Extremely  bad  Very  bad  Bad  Neutral  Good  Very  good  Extremely 

good 


MEAN  RESPI 

DNSE 

TELE 

SEMI 

AUTO 

4.63 

5.11 

5.63 

Comments 

TELE 


BUILDING  RECONNAISSANCE 


Useful  and  probably  more  practical;  however,  it  is  difficult  to  do  as  one 
person. 

I  prefer  the  control  of  the  robot,  but  ability  to  map  was  difficult. 

Having  to  drive,  map,  and  answer  questions  was  too  much. 


No.  of 
Responses 


1 

1 

1 
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No.  of 

Comments  Responses 

When  in  full  tele  mode,  it  is  hard  to  concentrate  on  main  mission  while 
having  to  react  to  outside  influences.  With  this,  frustration  comes  in 
making  tasks  seem  even  more  difficult.  1 

The  controls  are  hard  to  turn.  1 

Fixed  camera  may  be  a  problem.  1 

Rear  camera  would  be  nice  as  well.  1 

Robot  if  tactically  possible  could  sit  higher  from  ground  level  or  the 
camera.  1 

SEMI 

Allowed  focus  to  be  shifted  to  the  secondary  objective  while  maintaining 
the  primary  objective.  1 

Like  playing  with  a  toy/Xbox.  1 

I  had  a  tendency  to  want  to  have  more  control  over  the  robot.  1 

Rough  control  over  the  robot.  1 

I  didn’t  like  it  at  all.  I’d  rather  use  Tele  where  I  am  in  total  control.  But, 
of  course,  I  believe  that  would  be  subject  to  change  if  I  had  more 
practice  with  it.  1 

Robot  overlooked  nearly  an  entire  room,  and  got  itself  stuck  against  a 
wall.  Definitely  had  to  navigate  some  with  remote;  very  simple.  1 

This  level  is  worthless  to  me  and  the  camera  could  be  better  with  clarity 
and  the  ability  to  zoom  in  and  out.  1 

Not  as  aware  of  surroundings.  1 

AUTO 

Robot  did  its  job  very  well  allowing  me  to  add  focus  to  secondary 
objectives  without  compromising  my  primary  goal.  1 

This  mode  makes  it  easier  than  the  other  two.  It  allows  you  to  multitask 
a  lot  easier  than  the  other  two.  1 

Autonomous  mode  seems  to  be  the  way  to  go.  1 

Very  good.  Can  I  come  back  and  do  this  tomorrow?  1 

I  think  the  camera  should  be  providing  a  better  resolution  and  should  be 
moveable.  1 
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Intentionally  left  blank. 
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Appendix  E.  End  Of  Experiment  Questionnaire 


This  appendix  appears  in  its  original  form  without  editorial  change. 
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SAMPLE  SIZE  =  30 


1 .  Using  the  scale  below,  please  rate  the  following  characteristics  of  the  control 
system  that  you  used. 

1  2  3  4  5  6  7 

Extremely  bad  Very  bad  Bad  Neutral  Good  Very  good  Extremely  good 


CHARACTERISTICS 

MEAN  RESPONSE 

a.  Resolution  (clarity)  of  the  display 

4.63 

b.  Size  of  objects  appearing  in  the  display 

4.90 

c.  Ability  to  adjust  display 

3.83 

d.  Comfort  of  viewing  the  display 

4.30 

e.  Display  brightness 

4.77 

f.  Effect  of  glare  on  the  display 

4.03 

g.  Contrast  between  objects  on  the  driving  display 

4.93 

h.  Display  color 

4.13 

i.  Comfort  of  using  the  display 

4.73 

j.  Number  of  controls 

5.23 

k.  Control  locations 

5.43 

1.  Size  of  individual  controls 

5.67 

m.  Complexity  of  controls 

5.40 

n.  Ability  to  use  controls  without  inadvertently  activating  other 
controls 

5.60 

o.  Size  of  entire  control  unit 

5.80 

p.  Adequacy  of  this  control  unit  for  teleoperating  a  robot 

5.60 

q.  Adequacy  of  this  control  unit  for  semi-autonomousiy  operating 
a  robot 

5.67 

r.  Adequacy  of  this  control  unit  for  autonomously  operating  a 
robot 

5.87 

s.  Overall  assessment  of  this  control  unit 

5.40 

No.  of 

Comments  Responses 

The  map  display  overhead  was  excellent.  The  robot  updated  the 
mapping  of  the  room  fast  and  accurately.  1 

I  liked  the  familiarity  of  the  Xbox  controller.  1 

Display  was  small  and  lack  of  optics.  1 

The  most  difficult  part  was  identifying  objects  being  displayed  by  the 
camera.  1 

Picture  quality  wasn’t  too  good.  Couldn’t  tell  what  some  items  were.  1 

Depth  perception  on  display  was  hard  to  determine.  1 

The  lack  of  clarity  and  color  sometimes  caused  me  to  see  a  splotch  on 
the  low  part  of  a  wall  or  floor  and  think  it  was  an  object.  1 

The  display  would  be  better  if  it  was  in  color.  It  would  be  easier  to  ID 
clothing,  bloody  rags,  etc.  1 


48 


No.  of 

Comments  Responses 

Had  trouble  reading  maps/writing  on  the  wall.  1 

More  time  to  practice  would  have  played  a  bigger  part  in  rating  this  unit 
higher.  1 

2.  Please  indicate  your  1st,  2nd  and  3rd  choice  of  the  level  of  autonomy  to  use  for 
completing  the  following  tasks  by  placing  a  1 , 2,  or  3  in  the  appropriate  column. 


MEAN  RATING 

Tele¬ 

operation 

Semiautonomous 

operation 

Autonomous 

operation 

Route  planning 

1.62 

2.04 

2.35 

Generating  goals 

1.88 

1.96 

2.20 

Driving  a  straight 
route 

1.96 

2.15 

1.85 

Avoiding  obstacles 

1.81 

2.04 

2.15 

Looking  for  objects  of 
interest 

2.38 

1.96 

1.81 

Overall  building 
reconnaissance 

1.73 

1.96 

2.31 

3.  What  suggestions  do  you  have  for  ways  to  increase  the  effectiveness  of  the 


following  modes? 

No.  of 

Comments  Responses 

Teleoperation 

I  liked  tele  because  I  was  in  control;  but  I  wasn’t  a  fan  of  the  mapping.  1 

Tele  mode  offers  more  freedom  and  choice  when  operating  the  robot, 
but  it  takes  too  much  time.  1 

Easy  to  operate  but  you  have  to  do  everything  manual  so  it  is  a  little 
more  time  consuming.  1 

Side/rear  cameras.  1 

Robot  still  has  the  ability  to  map.  1 

Symbols  on  control  pad  to  show  what  it  will  do  (photos).  1 

Improve  lag.  3 

Adjust  screen  so  that  visibility  is  clearer.  1 

Wide  angle  screen.  1 

Being  able  to  use  the  laser  rangefinder.  1 

Better  zoom  on  camera.  1 

Better  camera  clarity  and  angle  of  view.  1 

Hot  keys  to  quickly  mark/label  items  of  interest.  1 

Color  monitor.  1 

Fewer  distractions.  1 

Automatically  go  to  label  a  picture  after  one  is  taken.  1 

The  controls  for  picture  taking  and  being  able  to  label  were  confusing  1 
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No.  of 

Comments  Responses 

after  a  while. 

Better  nav  controls.  1 

Improve  signal  for  response.  2 

Less  questions.  1 

Semiautonomous 

Works  well.  1 

More  responsive  to  user  suggestions  while  driving.  1 

Responsiveness  could  be  slow,  from  switching  in  and  out  of  semi  mode. 

Got  stuck  at  one  point.  1 

Make  more  like  autonomous  where  it  will  scan  the  room  while  moving.  1 

Robot  identifies  objects  that  are  inconsistent  with  surroundings  more 
clearly.  1 

Use  to  ID  objects.  1 

Lower  speed  to  stop  for  pictures.  1 

Improve  lag.  2 

Better  zoom  on  camera.  1 

Hot  keys  to  quickly  mark/label  items  of  interest.  1 

Get  rid  of  it  because  you  spend  more  time  in  the  tele  mod.  1 

Color  monitor.  1 

Wide  angle  screen.  1 

Did  not  like  because  it  was  like  a  retard  robot.  1 

Symbols  on  control  pad  to  show  what  it  will  do  (photos).  1 

Could  have  a  temp  pause  button  instead  of  two  (one  to  stop  and  one  to 
resume).  1 

Autonomous 

This  is  the  best  mode.  1 

Worked  great!  1 

Excellent  for  the  job  and  really  simple.  1 

Works  well.  1 

Speedy.  1 

Full  room  scans  via  camera.  1 

Color  monitor.  1 

Wide  angle  screen.  1 

Better  zoom  on  camera.  1 

Hot  keys  to  quickly  mark/label  items  of  interest.  1 

Screen  clarity.  1 

Make  sure  the  robot  travels  to  every  corner  of  the  room  so  the  operator 
can  see  everything  before  it  moves  on  to  the  next  room.  1 

Limits  the  effectiveness  of  how  well  you  can  conduct  recon  on  a 
building.  1 

Symbols  on  control  pad  to  show  what  it  will  do  (photos).  1 

Help  robot  navigate  around  objects  easier.  1 

Time  delay.  1 

More  practice  on  it.  1 
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4.  Using  the  scale  below,  please  rate  the  level  of  reliability  (trust)  of  the  autonomous 
systems  you  used. 


1  2  3  4  5 

Extremely  Very  Unreliable  Neutral  Reliable 

unreliable  unreliable 


6  7 

Very  Extremely 

reliable  reliable 


MEAN  RESPONSE 

Semi-autonomous 

5.43 

Autonomous 

5.68 

No  comments. 


5.  Using  the  scale  below,  please  rate  how  helpful  the  following  capabilities  were  to  your 
ability  to  drive/control  the  robot  in  the  semi-autonomous  and  autonomous  modes. 


1  2  3  4  5  6  7 

Extremely  very  unhelpful  neutral  helpful  very  extremely 

unhelpful  unhelpful  helpful  helpful 


CAPABILITY 

MEAN  RESPONSE 

a.  Robot’s  position/location 

5.57 

b.  Showing  the  path  the  robot  drove 

5.80 

c.  Obstacle  avoidance 

5.53 

d.  Ability  to  return  to  “start  location” 

6.27 

e.  Map  data  (robot  built) 

6.11 

f.  Capability  of  the  robot  to  drive  itself 

6.00 

6.  Would  you  want  the  following  capabilities  again  if  you  have  to  perform  these  same 
missions  over  again? 


NUMBER  OF  RESI 

PONSES 

YES 

NO 

NR 

a.  Robot’s  position/location 

25 

1 

4 

b.  Showing  the  path  the  robot  drove 

25 

1 

4 

c.  Obstacle  avoidance 

26 

0 

4 

d.  Ability  to  return  to  “start  location” 

25 

1 

4 

e.  Map  data  (robot  built) 

24 

0 

6 

f.  Capability  of  the  robot  to  drive  itself 

26 

0 

4 
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7.  How  would  you  improve  the  robot  and/or  any  of  its  features? 


No.  of 

Comments  Responses 

Better  camera  with  higher  resolution.  7 

Improve  clarity  or  provide  light  source  of  some  kind.  2 

Bigger  camera  view.  3 

Camera  positioning  and  control.  1 

Moveable  camera  with  zoom  feature.  2 

Add  a  wider  lens  that  elevates  and  swivels  for  easier  identification.  1 

An  arm  and  elevation  on  camera.  1 

Add  a  scanning  laser  that  identifies  objects  as  question  marks  on  the 
screen  in  the  auto  mode.  1 

Constant  stopping  and  adjusting  view  in  the  tele  mode  for  pictures  took 
lots  of  time.  Maybe  add  in  a  “room  scan”  so  the  robot  automatically 
visually  covers  whole  room.  1 

Photos  can  be  pop-up  windows  (1  -1 0),  then  you  can  label  from  there 
instead  of  having  a  picture  in  the  middle  of  your  pathway  on  the  display.  1 

Adjust  photo  movement  (up  and  down).  1 

If  the  robot  would  recognize  and  take  photos  on  its  own.  1 

Add  an  extension  for  the  lens.  1 

Lag  time  was  main  problem  and  it  got  confused  sometimes  concerning 
where  it  was  going;  maybe  make  an  adjustment  for  that.  1 

Avoiding  obstacles  a  little  better  in  semi  and  auto  modes.  1 

The  avoidance  feature  needs  to  be  able  to  be  deactivated.  1 

Map  data  needs  to  be  clearer.  1 

Improve  the  elevation  view  and  the  capability  to  climb  stairs  and  not  get 
stuck.  1 

Height  of  robot.  1 

Speed.  1 

Turning  ability.  1 

Make  it  lighter  so  it  would  be  easier  to  transport  and  add  a  light  to 
illuminate  objects  that  are  in  the  dark.  1 

Quieter.  1 


8.  How  would  you  improve  the  interface  and/or  controller? 


No.  of 

Comments  Responses 

Already  good.  2 

Very  easy  to  understand;  leave  as  is.  5 

Most  soldiers  are  into  games,  so  that  type  of  controller  works  in  this 
situation.  1 

Perfect  controller.  2 

Different  control.  1 

Add  hot  keys  to  quickly  mark/tag  items  of  interest.  1 
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No.  of 


Comments  Responses 

Add  a  button  for  elevation  scan.  1 

Add  a  warning  LED  when  in  tele  to  warn  the  user  of  objects  that  they  will 
hit.  1 

Add  an  instrument  panel  that  shows  distance  cleared,  total  distance 
traveled,  height  of  the  room,  etc.  1 

Improve  lag  on  display.  1 

Identify  objects  on  map  more  clearly.  1 

Video  display  should  take  up  about  half  of  the  screen.  When  a  lot  of 
objects  were  nearby,  it  was  difficult  to  tell  one  picture  from  another. 

The  labels  should  be  in  alphabetical  order  when  they  are  displayed.  1 

Faster  mouse  speed.  1 

Asking  for  a  label  automatically  whenever  a  picture  is  taken.  2 

The  X,Y,A,B  buttons  would  be  better  for  selecting  items,  taking  pictures, 
zoom  in  and  out.  These  are  the  closest  buttons  to  your  hand  and  the 
most  used.  The  LB  and  RB  are  the  least  used  in  Xbox  games  for  that 
reason.  They  are  inconveniently  placed  and  awkward.  1 

Alternate  controls  (dual-stick  tank  control).  1 

It  is  easier  to  view  with  right  thumb  stick  left/right  and  move  with  the  left 
thumb  stick.  1 

Perhaps  360°  viewing  ability.  2 

Get  more  input  from  gamer  for  functionality.  1 

Bigger  joystick  with  movement  buttons  on  it  instead  of  controller  front.  1 

Larger  screen  for  room/path  image.  1 

Wide  angle  lens.  1 

Color.  1 

Zoom  feature  on  the  video  screen  bottom;  sort  of  like  a  drop-down  menu 
when  you  change  the  size  of  your  pictures  on  your  PC  when  you  print.  1 


9.  a.  Did  you  feel  a  difference  between  the  semi-autonomous  and  autonomous  modes? 

9  No 
21  Yes 

b.  If  yes,  what  was  the  difference(s)? 

Comments 
Auto 

The  auto  mode  will  go  into  a  room  and  scan  the  room  for  you  giving  you 
more  time  to  look  for  things. 

Lets  you  go  from  A  to  B  and  work  backwards  from  B  to  A. 

Seems  to  return  to  the  starting  position  more  quickly  and  efficiently. 

Ease  of  operation  and  speed. 

More  aware. 

Maps  better. 


No.  of 


1 

1 

1 

2 

1 

2 
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No.  of 


Comments 

Easier  to  focus  on  what  to  take  photos  of,  instead  of  doing  both. 

Total  tech  control. 

Recons  the  site  faster. 

Semi 

Semi  mode  moved  distances  faster. 

The  robot  moving  itself. 

Ease  of  movement  in  between  objects. 

Semi  seemed  to  be  smoother  with  the  controls. 

More  control  (without  stopping). 

Obstacle  avoidance  and  new  unidentified  exploring. 

User  input  during  semi  mode  was  helpful. 

You  pay  attention  to  making  corrections  and  having  to  over  compensate 
Messed  up  (got  stuck,  lagged  -  not  a  fan!). 

Does  not  seem  as  aware  as  auto. 

General 

The  difference  was  hard  to  discern  because  I  was  stopping  the  robot 
just  as  often  in  both  to  take  pictures. 

10.  a.  Which  mode  do  you  prefer? 

16  Autonomous 
14  Semi-autonomous 

b.  What  are  the  reasons  for  your  preference? 

Comments 


Autonomous 

Didn’t  lag,  ran  smoother.  I  never  got  stuck.  Felt  like  I  could  look  at 
OPORDER  and  not  worry  if  it’d  get  stuck  or  confused.  1 

Easier  to  work  with  because  it  goes  to  every  corner  of  the  building.  1 

Easy  to  operate  and  I  like  to  see  the  area  with  everything  in  it.  1 

More  comprehensive  and  fully  maps  building.  1 

Ability  to  focus  more  on  the  screen  than  controlling.  1 

Allows  me  to  concentrate  on  objects  that  can  make/break  the  mission 
success.  1 

Ease  of  operation  and  speed.  1 

Lets  you  multitask  easier.  3 

Seemed  to  know  where  it  was  going.  1 

I  don’t  really  get  the  difference  between  the  two.  I  understand  what  it’s 
supposed  to  do,  but  mostly  I  just  stopped  it  and  ran  it  on  my  own.  1 

Semi-autonomous 

Ability  to  guide  robot  during  travel.  1 

It  still  seemed  necessary  due  to  the  number  of  items  and  also  the  robot 
started  to  skip  past  a  room.  Although  it  may  have  eventually  gone  1 


1 

1 

1 


1 

1 

1 

1 

3 

1 

1 

1 

1 

1 
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No.  of 

Comments  Responses 

back,  it  seemed  more  efficient  to  just  take  control. 

Semi  is  faster  at  distance  travel.  1 

I  like  the  interactive  control.  1 

I  feel  I  have  more  and  faster  control  of  the  robot.  5 

Moving  on  its  own  and  still  having  control.  1 

Easy  to  change  course  slightly.  1 

Best  feature  is  the  robot  driving  itself.  Therefore,  they  were  both  a  good 
feature.  1 

Exploring  something  quickly  and  laying  out  the  floor  plans.  1 

Manual  control  is  preferred.  1 


1 1 .  In  the  semi-autonomous  and  fully-autonomous  modes,  the  robot  automatically  built 
a  floor  plan  of  the  building  for  you.  How  often  did  you  use  the  floor  plan  as  you  looked 
for  objects  of  interest?  How  was  it  helpful  or  not  helpful? 


No.  of 

Comments  Responses 

Very  often.  Absolutely  helpful.  7 

It  helped  me  know  where  I  was  at  all  times.  1 

I  constantly  viewed  the  mapped  floor  plan  to  know  where  the  robot  had 
gone  and  where  it  needed  to  go.  1 

Allowed  for  easy  positioning,  along  with  mission  check.  1 

Map  reading  and  using  a  map  may  seem  out  of  date;  however,  it  is  of 
great  importance  to  the  Infantry.  1 

Probably  the  most  important  tool.  I  used  it  to  see  where  the  robot  had 
been  and  what  pictures  I’d  taken,  areas  I  needed  to  search,  and  the 
route  I  took.  It  was  a  very  good  and  important  asset  to  the  operation.  1 


I  used  the  floor  plan  just  to  see  the  parameters  of  the  building  and  to  see 


where  the  robot  was  in  the  building  initially.  But  I  could  have  figured 
these  things  out  myself.  However,  the  floor  plan  was  helpful  because  it 
was  an  added  visual  that  showed  where  the  robot  had  been  in  the 
building  during  the  recon.  1 

80  percent  of  the  time.  Allowed  you  to  scan  for  items  or  interest.  1 

Used  it  often  to  find  out  where  I  was  and  where  I  needed  to  look.  2 

Used  often  to  check  for  missed  doors.  1 

It  was  helpful,  but  didn’t  use  it  very  much.  1 

It  helped  more  with  the  basic  area  to  work  in.  1 

You  knew  where  you  have  already  been  as  far  as  room  clearing.  It  was 
helpful  in  that  manner.  2 

Good  for  reference  to  whether  photographed  object  yet  or  not.  1 

I  used  it  to  make  sure  I  didn’t  miss  anything  and  rooms.  When  I  was  in 
full  control,  I  worked  the  rooms  clockwise  making  sure  it  was 
completely  cleared  before  moving  on,  and  then  continued.  When  done, 

I  went  straight  back  to  the  beginning  by  pressing  the  return  home  1 
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No.  of 

Comments  Responses 

button. 

I  didn’t  really  use  it  other  than  to  tell  if  I  had  already  been  in  that  room. 

Semi-helpful.  1 

Somewhat  helpful.  The  area  being  explored  was  small,  so  the  mapping 
feature  was  less  important.  However,  in  a  large  building  the  mapping 
feature  would  be  extremely  useful.  1 

It  was  helpful  until  it  started  wandering  aimlessly  without  completing  the 
course.  The  floor  plan  confused  me.  1 

It  was  helpful  but  not  fresh  in  the  room.  Maybe  to  have  an 
understanding  for  other  sources  or  reasons.  For  example,  the  robot 
takes  pictures  and  has  a  floor  plan.  Now  another  group  can  come  in  to 
collect  intel  knowing  already  what’s  there  getting  themselves  in  it.  1 

Not  very  helpful  at  first,  but  as  I  got  more  comfortable,  the  usefulness 
increased.  1 

Sometimes  I  spent  more  time  looking  at/for  objects  and  determining 
what  they  were.  1 

I  did  not  use  it  much  to  find  objects;  however,  the  map  is  incredibly 
useful  for  site  recon  and  future  operations.  It  gives  a  much  more 
accurate  assessment  than  I  could  draw  using  the  display.  1 

I  only  used  the  floor  plan  to  go  back  and  make  sure  I  didn’t  miss  any 
hidden  items.  1 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


ACS 

Autonomous  Capability  Suite 

ANOVA 

analysis  of  variance 

ARL 

U.S.  Army  Research  Laboratory 

COTS 

commercial  grade  off-the-shelf 

GPS 

global  positioning  systems 

HRED 

Human  Research  and  Engineering  Directorate 

IMU 

inertial  measurement  unit 

MOCU 

Multi-Robot  Operator  Control  Unit 

MOUT 

military  operations  in  urban  terrain 

NASA-TLX 

National  Aeronautics  and  Space  Administration-Task  Load  Index 

OCS 

Officer  Candidate  School 

POW 

prisoner  of  war 

SA 

situation  awareness 

SSC  Pacific 

Space  and  Naval  Warfare  Systems  Center  San  Diego 

SUGV 

small  unmanned  ground  vehicle 

WTC 

Warrior  Training  Center 
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NO.  OF 

COPIES  ORGANIZATION 


1  DEFENSE  TECHNICAL 
(PDF  INFORMATION  CTR 
only)  DTIC  OCA 

8725  JOHN  J  KINGMAN  RD 
STE  0944 

FORT  BEL  VOIR  VA  22060-6218 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
IMNE  ALC  HRR 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
RDRL  CIM  L 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
RDRL  CIM  P 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
RDRL  D 

2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 


ABERDEEN  PROVING  GROUND 

1  DIR  USARL 

RDRL  CIM  G  (BLDG  4600) 
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NO.  OF 

COPIES  ORGANIZATION 


NO.  OF 

COPIES  ORGANIZATION 


1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  A  I  MARTIN 
MYER  CENTER  BLDG  2700  RM  2D31 1 
FORT  MONMOUTH  NJ  07703-5601 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  C  A  DAVISON 
320  MANSCEN  LOOP  STE  1 15 
FORT  LEONARD  WOOD  MO  65473 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  DI  T  DAVIS 
BLDG  5400  RM  C242 
REDSTONE  ARSENAL  AL  35898-7290 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRS  EA  DR  V  J  RICE 
BLDG  4011  RM  217 
1750  GREELEY  RD 
FORT  SAM  HOUSTON  TX  78234-5002 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  DG  R  SPINE 
BLDG  333 

PICATINNY  ARSENAL  NJ  07806-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
ARMC  FIELD  ELEMENT 
RDRL  HRM  CH  C  BURNS 
THIRD  AVE  BLDG  1467B  RM  336 
FORT  KNOX  KY  40121 

1  ARMY  RSCH  LABORATORY  -  HRED 
AWC  FIELD  ELEMENT 
RDRL  HRM  DJ  D  DURBIN 
BLDG  4506  (DCD)  RM  107 
FORT  RUCKER  AL  36362-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  CK  J  REINHART 
10125  KINGMAN  RD 
FORT  BEL  VOIR  VA  22060-5828 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AY  M  BARNES 
2520  HEALY  AVE 
STE  1172  BLDG  51005 
FORT  HUACHUCA  AZ  85613-7069 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HR  MP  D  UNGV ARSKY 
POPE  HALL  BLDG  470 
BCBL  806  HARRISON  DR 
FORT  LEAVENWORTH  KS  66027-2302 


1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AJ  J  HANSBERGER 
JFCOM  FE 

1 15  LAKE  VIEW  PKWY  STE  B 
SUFFOLK  VA  23435 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  DQ  MR  FLETCHER 
NATICK  SOLDIER  CTR 
AMSRD  NSC  WS  E  BLDG  3  RM  343 
NATICK  MA  01760-5020 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AT  J  CHEN 
12350  RESEARCH  PKWY 
ORLANDO  FL  32826-3276 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AT  C  KORTENHAUS 
12350  RESEARCH  PKWY 
ORLANDO  FL  32826 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AS  C  MANASCO 
SIGNAL  TOWERS 
BLDG  29808A  RM  303A 
FORT  GORDON  GA  30905-5233 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  CU 
6501  E  1 1  MILE  RD  MS  284 
BLDG200A  2ND  FL  RM2104 
WARREN  MI  48397-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
FIRES  CTR  OF  EXCELLENCE 
FIELD  ELEMENT 
RDRL  HRM  AF  C  HERNANDEZ 
3040  NW  AUSTIN  RD  RM  221 
FORT  SILL  OK  73503-9043 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AV  S  MIDDLEBROOKS 
91012  STATION  AVE  RM  348 
FORT  HOOD  TX  76544-5073 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  CN  R  SPENCER 
DCSFDI  HF 

HQUSASOC  BLDG  E2929 
FORT  BRAGG  NC  28310-5000 
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NO.  OF 

COPIES  ORGANIZATION 


1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  DW  E  REDDEN 
BLDG  4  CL  60 

FORT  BENNING  GA  31905-5400 

1  ARMY  G1 
(CD  DAPE  MR  B  KNAPP 
only)  300  ARMY  PENTAGON  RM  2C489 
WASHINGTON  DC  20310-0300 


ABERDEEN  PROVING  GROUND 

5  DIR  USARL 
RDRL  CIM  G 
S  FOPPIANO 
RDRL  HR 
L  ALLENDER 
T  LETOWSKI 
RDRL  HRM  B 
I  LOCKETT 
RDRL  HRS  D 
B  AMREIN 
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